-- Logs begin at Tue 2022-10-04 07:54:14 UTC, end at Tue 2022-10-04 09:02:58 UTC. -- Oct 04 08:20:15.186600 dv-cinder-860209-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 04 08:20:15.194242 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Oct 4 08:20:15 2022] *** Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: nodename: dv-cinder-860209-1-iscsi Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: machine: x86_64 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: clock source: unix Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: pcre jit disabled Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: detected number of CPU cores: 4 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: current working directory: / Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: detected binary path: /usr/bin/uwsgi-core Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: your processes number limit is 47680 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: your memory page size is 4096 bytes Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: detected max file descriptor number: 2048 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: lock engine: pthread robust mutexes Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: thunder lock: enabled Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 04 08:20:15.197867 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 04 08:20:15.237806 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: Python main interpreter initialized at 0x55a9264cf9c0 Oct 04 08:20:15.237806 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: python threads support enabled Oct 04 08:20:15.237806 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: your server socket listen backlog is limited to 100 connections Oct 04 08:20:15.237806 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: your mercy for graceful operations on workers is 90 seconds Oct 04 08:20:15.238188 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: mapped 403077 bytes (393 KB) for 2 cores Oct 04 08:20:15.238265 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: *** Operational MODE: preforking *** Oct 04 08:20:15.238511 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: *** uWSGI is running in multiple interpreter mode *** Oct 04 08:20:15.238511 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: spawned uWSGI master process (pid: 81390) Oct 04 08:20:15.238743 dv-cinder-860209-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 04 08:20:15.239853 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: spawned uWSGI worker 1 (pid: 81404, cores: 1) Oct 04 08:20:15.239853 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: spawned uWSGI worker 2 (pid: 81405, cores: 1) Oct 04 08:20:15.239853 dv-cinder-860209-1-iscsi devstack@keystone.service[81390]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 04 08:20:19.553070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:20:19.553070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: from cryptography.utils import int_from_bytes Oct 04 08:20:19.553829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:20:19.553829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: from cryptography.utils import int_from_bytes Oct 04 08:20:19.555355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:20:19.555355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: from cryptography.utils import int_from_bytes Oct 04 08:20:19.555815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 04 08:20:19.555815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: from cryptography.utils import int_from_bytes Oct 04 08:20:19.604054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 08:20:19.604271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 08:20:19.604418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] command line args: [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 08:20:19.604681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 08:20:19.604854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ================================================================================ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 08:20:19.605104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] admin_token = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.605343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_dir = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.605526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.605703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_source = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.605822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 04 08:20:19.605907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] debug = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.606015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 04 08:20:19.606100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.606318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] command line args: [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 04 08:20:19.606448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.606531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 04 08:20:19.606632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.606699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ================================================================================ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 04 08:20:19.606859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.606939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] admin_token = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_dir = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607372 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_source = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_config_append = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.607656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] debug = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_dir = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.608920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_config_append = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_dir = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.609984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.610956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.611526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.612155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.612320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.612556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.612782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.612922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] publish_errors = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.613862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] standard_threads = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] publish_errors = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.614923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_journal = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_json = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_stderr = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_syslog = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] standard_threads = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.615879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.616134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.616448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_journal = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.616629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_json = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.616836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_stderr = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.616961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_syslog = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.617074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 04 08:20:19.617273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.617921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.618971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.619962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.620853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.621968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.622951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.623913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.624922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.625987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.626992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.627972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.628962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.629939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630364 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.630950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.631953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.632888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.633960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.634931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.635933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.636866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.637936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638380 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.638949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.639853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.640934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.external = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.password = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.641962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.token = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.external = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.642985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.password = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.643975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.token = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.644870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.645968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.646950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.647994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648380 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.648943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.649888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.650856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.651944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.652957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.653872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.654985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.655999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.656933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.657847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.658003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.658088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.658240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.658319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.659930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.660993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.661890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.662905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.663877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.664937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.665862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.666995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.667921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.668945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.669877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.670985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.671991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.672942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.673866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.674091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.674798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.674928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.675996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.676937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.677979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.678972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.679893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.680943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.681914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.682979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.683896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.684051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.684138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.684299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.684379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.688592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.689253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.690348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.692952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.693965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.694953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.695956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.696962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 04 08:20:19.697168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 04 08:20:19.722131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.722347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.722429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.722567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.722655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.722864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.731004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.731206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.759866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.760087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.760567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.760799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.814166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.815156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.864511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.864750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.868100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:19.868334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.023972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.024103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.024189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.024281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.024392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.024604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.047904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.048668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.060253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.060626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.073954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.074627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 04 08:20:20.092152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.092327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.092408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:20:20.092515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:20:20.092689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.092862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.092951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:20:20.093058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 04 08:20:20.093141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.093919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.094092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.103820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.103929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.104725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.112425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.112532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.115343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.115434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.115625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.115707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.115905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.116997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.117961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.118090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.118220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.140467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.140664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.141926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.142089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.142230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.142386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.147612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.147890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.148022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.148150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.148290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.148424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.154432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.154780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.154950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.155076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.161126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.161417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.161561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.161723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.161862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.161997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.162832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.163040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.163183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.163485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.163795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.178414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.178709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.179397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.179672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.182469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.183498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.185639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.185929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.186704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.186993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.190190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.191260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.193259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.194428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.196431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.196729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.197959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.198182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.198464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.198768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.199052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.213984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.214284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.214534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.214815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.215328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.215612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.215873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.216129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.222344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.223482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.224040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.224996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.225573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.225847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.226519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.226829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.228513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.228793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.229072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.229349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.229505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.229738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.229820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.230049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.230323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.230739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.241271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.241548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.241822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.242177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.242955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.243037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.243325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.243611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.243885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.258339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.259317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.261253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.262199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.264122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.265095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.265587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.266520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.267079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.267359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.267757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.268586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.268924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.269323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.274705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.275659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.276159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.277108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.279044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.279433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.279716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.279992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.280073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.280389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.280673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.280950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.287925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.288218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.288499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.288838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.289124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.289411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.295373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.295650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.295924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.296197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.296298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 04 08:20:20.296470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.296571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.296740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.296824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.297908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.298920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.299036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.299335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.299601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.299872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 04 08:20:20.333324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55a9264cf9c0 pid: 81405 (default app) Oct 04 08:20:20.334240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55a9264cf9c0 pid: 81404 (default app) Oct 04 08:20:20.336588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b4f4-2138-4bcb-9c5c-407225f24f0b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:20.336872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b4f4-2138-4bcb-9c5c-407225f24f0b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:20.337140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0124b4f4-2138-4bcb-9c5c-407225f24f0b None None] PATH_INFO: `/v3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:20.341250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1/1] 10.222.0.16 () {50 vars in 791 bytes} [Tue Oct 4 08:20:20 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:27.172319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:27.172805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:27.172890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:27.178191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1/2] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:27 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:27.183640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:27.184163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:27.184163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:27.255289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: INFO dbcounter [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] Registered counter for database keystone Oct 04 08:20:27.274122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG oslo_db.sqlalchemy.engines [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81405) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 08:20:27.280372 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writer thread running {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 08:20:27.460908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG passlib.handlers.bcrypt [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81405) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 08:20:27.461207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG passlib.handlers.bcrypt [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81405) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 08:20:27.519617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:27.521242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:27.650333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 2/3] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 468 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 04 08:20:28.083584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:28.154099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: INFO dbcounter [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] Registered counter for database keystone Oct 04 08:20:28.177006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG oslo_db.sqlalchemy.engines [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81404) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 04 08:20:28.183346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writer thread running {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 04 08:20:28.304449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb6ca2-9468-4722-b2c1-620e19d3c52d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:28.308699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:28.309325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:28.309600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:28.309875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:28.318825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:28.370300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING keystone.common.rbac_enforcer.enforcer [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 08:20:28.370589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bf1aca-d6ed-4a29-81b4-64090ddc7ee1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:28.380462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 2/4] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Oct 4 08:20:28 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 08:20:28.386419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:28.403199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afcf30c-2138-4fe7-9ae7-71b204c804c2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:28.407556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:28.408663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:28.408941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:28.409742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] PATH_INFO: `/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:28.411339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] RBAC: Authorizing `identity:get_project(project_id=dc7bdc31e1404eecb2eaeb287aaeab06)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:28.464019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING keystone.common.rbac_enforcer.enforcer [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 04 08:20:28.464373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef84ded8-c885-4e7a-9bfe-a04892873e08 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:28.469172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 3/5] 10.222.0.16 () {56 vars in 1207 bytes} [Tue Oct 4 08:20:28 2022] GET /identity/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06 => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:30.646679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59561cad-ad73-4a41-9653-47120c95c0f9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:30.647238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59561cad-ad73-4a41-9653-47120c95c0f9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:30.647238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59561cad-ad73-4a41-9653-47120c95c0f9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:30.651321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 3/6] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:30.657001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:30.657435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:30.657552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:30.678501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:30.680041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:30.687767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 4/7] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 04 08:20:30.956822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59561cad-ad73-4a41-9653-47120c95c0f9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:30.968839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59561cad-ad73-4a41-9653-47120c95c0f9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:30.973220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:30.973824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:30.974107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:30.974384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:30.982182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:30.984570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4982af-21e5-4f3a-8c49-511cefb3ef98 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:30.997177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 4/8] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Oct 4 08:20:30 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:20:31.002956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:31.012562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93e81eb-fa4a-4888-90e4-43e06b564c15 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:31.016902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:31.017550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:31.017825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:31.018092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:31.019598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:31.021421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceefce39-c39a-4718-869a-dc2b09bd3e6a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:31.022972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 5/9] 10.222.0.16 () {56 vars in 1198 bytes} [Tue Oct 4 08:20:31 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:33.272227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91b2bd-e5a9-45b9-a745-e4828fd2df9e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.272770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91b2bd-e5a9-45b9-a745-e4828fd2df9e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.272770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91b2bd-e5a9-45b9-a745-e4828fd2df9e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.274086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 5/10] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:33.279486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.279941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.280044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.297962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:33.299174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:33.306325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0f13a-46c4-42bf-b8d7-47f61487a6d9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.306628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0f13a-46c4-42bf-b8d7-47f61487a6d9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.306901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0f13a-46c4-42bf-b8d7-47f61487a6d9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.308230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 6/11] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:33.314565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.314855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.315130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.369873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG passlib.handlers.bcrypt [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81404) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 04 08:20:33.369873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG passlib.handlers.bcrypt [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81404) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 04 08:20:33.389056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 6/12] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:33.418321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:33.419883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:33.520772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 7/13] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:33.663747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:33.688818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:33.775829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8722c8-9fa0-4d44-9d8b-9f1a71ee89bb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:33.786999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:33.810684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:33.811311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.811593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.811870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.823045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:33.832635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:33.833581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:33.834382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 7/14] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:20:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:33.841142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:33.889020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fadc4bb-220b-4391-9e9d-f8c87e13898f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:33.891401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:33.927476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:33.927916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:33.928347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.928631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.928902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.947234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b095505-4272-472b-91fd-5a92462a6f88 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:33.981831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:33.982437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:33.982763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:33.983027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:33.983698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:33.984000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:33.985671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:33.998252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 8/15] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:33.998775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.000305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.003848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:34.103158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 8/16] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:20:34.103888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb4477-354e-451b-93af-f0d78081b282 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:34.108668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:34.111684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:34.139281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:34.139919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:34.140647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:34.140926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:34.142084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.143595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.145426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 9/17] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:34 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:34.150778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:34.174261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:34.214236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92bbd2-ba8f-4881-b49e-6e4af07ada96 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:34.249047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:34.249900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:34.250179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:34.250449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:34.261276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b646d2d-1f9f-4475-9358-9e07e21eb10a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:34.268890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.271116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:20:34.290707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:34.292022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:20:34.292811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:20:34.292811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:20:34.292811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:34.292811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 9/18] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:20:34.296806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:34.297440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:34.297561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:34.297712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:34.298001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:34.299126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.302084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.302927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 10/19] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:34.308657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:34.367470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:34.389555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:34.408172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cb6f-7f0f-479f-8c3f-b8e62fa9f177 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:34.414356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1a4593-b81b-4a4a-b764-07559e46642e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:34.442896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:34.443503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:34.443782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:34.444055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:34.444781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.domain.id=default, user.id=a17a7c6f52b74569b1779d66c4e1a9d1) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:34.445086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.446525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875272a4-b548-4a2a-a09f-3434caead701 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.449103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92252292-b023-463b-a178-7e913edc0221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:34.449715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92252292-b023-463b-a178-7e913edc0221 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:34.449999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92252292-b023-463b-a178-7e913edc0221 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:34.450273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92252292-b023-463b-a178-7e913edc0221 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:34.450948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92252292-b023-463b-a178-7e913edc0221 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:34.451248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92252292-b023-463b-a178-7e913edc0221 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:34.452620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92252292-b023-463b-a178-7e913edc0221 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:34.453329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 11/20] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Oct 4 08:20:34 2022] GET /identity/v3/role_assignments?user.id=a17a7c6f52b74569b1779d66c4e1a9d1&scope.domain.id=default&role.id=d7f1381a769e433aa1b033022931be96 => generated 233 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:34.472743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 10/21] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:20:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:36.971484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6dbf4-3c97-4560-b0c3-7795fb91752e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:36.972026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6dbf4-3c97-4560-b0c3-7795fb91752e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:36.972026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b6dbf4-3c97-4560-b0c3-7795fb91752e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:36.973937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 12/22] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:36.985990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a499f-cdac-4b63-b929-12716fd7c920 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:36.986524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a499f-cdac-4b63-b929-12716fd7c920 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:36.986524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a499f-cdac-4b63-b929-12716fd7c920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.015839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c44a499f-cdac-4b63-b929-12716fd7c920 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:37.017089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c44a499f-cdac-4b63-b929-12716fd7c920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:37.111911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 11/23] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:37.188685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29614c9-905f-4e2e-ac18-5b8c3cb54e01 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:37.189308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29614c9-905f-4e2e-ac18-5b8c3cb54e01 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:37.189806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29614c9-905f-4e2e-ac18-5b8c3cb54e01 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.191409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 13/24] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:37.197397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:37.197681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:37.197959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.227721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:37.228851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:37.232412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaaee9-be04-44b3-a137-64fdc618bbfe None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:37.232976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaaee9-be04-44b3-a137-64fdc618bbfe None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:37.233444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaaee9-be04-44b3-a137-64fdc618bbfe None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.235373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 14/25] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:37.248171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:37.248755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:37.249246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.283583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:37.285805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:37.296265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:37.335032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 12/26] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:37.424499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 15/27] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:37.619673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:37.715814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:37.762789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af5b692-06af-428d-804f-a6eed841b21b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:37.834197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:37.839458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:37.840188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:37.840679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:37.861640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:37.871670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:37.886607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:37.887494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:37.888266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:37.890666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 13/28] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:20:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:37.897690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:37.929354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae84a2-10d3-4058-b050-7aec2b709acc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:37.956714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.007644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.008266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.009024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.009455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.010305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.012195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef666f-f78d-4da2-8f47-cf9d8afa3fba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.057520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb859f33-cb9f-457f-878a-3b2f56c98fd0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.101362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.101984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.102263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.102536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.103239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:38.103542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.104926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.120531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 14/29] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:38.121458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.169025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 16/30] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Oct 4 08:20:37 2022] POST /identity/v3/roles => generated 219 bytes in 456 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:20:38.170838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97ac53-fbb6-43a0-b2eb-2a8865cfd5ea None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.171406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97ac53-fbb6-43a0-b2eb-2a8865cfd5ea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.171877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97ac53-fbb6-43a0-b2eb-2a8865cfd5ea None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.178444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 17/31] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:38.179800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97ac53-fbb6-43a0-b2eb-2a8865cfd5ea None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.294162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.300176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931e67f-3f9c-455c-a566-78e15c0efa88 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.385086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97ac53-fbb6-43a0-b2eb-2a8865cfd5ea None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.385611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.386497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.389050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.389585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.415566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.418088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ced5742a-3931-4b52-86c9-b431c1825356 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:20:38.439888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:38.440808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:20:38.442430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:20:38.442430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:20:38.442430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:38.442430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 15/32] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:38.443337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.443337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.443578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.447734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.448650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.449224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.449704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.451262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.453292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.455209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 18/33] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:38 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:38.456175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.460211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.469453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:38.470980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:38.558603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 16/34] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:38.558603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.603419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ec49a-86da-4a19-b655-4c991031051e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.645996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.659345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.660646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.661426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.661914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.662812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:38.663334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.665026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5de6b-08e9-4ed5-9dca-c52d053e1c1e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.669585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973392fd-f4ef-4b9f-9a22-d93bdc2a2a1d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.695328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 19/35] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:20:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:38.696992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550ee16-c479-4402-a108-7448d0c3d78f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.697498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550ee16-c479-4402-a108-7448d0c3d78f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.697987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550ee16-c479-4402-a108-7448d0c3d78f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.702573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 20/36] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:38.702573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550ee16-c479-4402-a108-7448d0c3d78f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.711408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.712067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.713101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.713101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.713993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.715502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40909a52-9d6c-47d8-bd1b-005a058bca3b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.717261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 17/37] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:38.719026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.719308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.719585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.740110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:38.741666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:38.795700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.819091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 18/38] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:38.820033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.845202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550ee16-c479-4402-a108-7448d0c3d78f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:38.916995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:38.917629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:38.917914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:38.918192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:38.918891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:38.927232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:38.956463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:38.993002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 21/39] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:20:38 2022] POST /identity/v3/projects => generated 301 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:38.994130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:38.996719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3527388-990d-48e5-9ea9-6349b133b327 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:39.073750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:39.074453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.074759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.075039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] PATH_INFO: `/v3/domains/default/users/a17a7c6f52b74569b1779d66c4e1a9d1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.077279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=a17a7c6f52b74569b1779d66c4e1a9d1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:39.078813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2805b0-79d9-47ab-b1f5-e27af23b29eb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:39.162914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3cf36d-ccba-4915-a754-3aa90b7dce23 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:39.181290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:39.206702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:39.207906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.208224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.208513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.209214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:39.209981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 19/40] 10.222.0.16 () {58 vars in 1361 bytes} [Tue Oct 4 08:20:38 2022] PUT /identity/v3/domains/default/users/a17a7c6f52b74569b1779d66c4e1a9d1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:20:39.210665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7c1923-78fa-4238-a592-8b2337d1d7fc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:39.211829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86bc9b-6a6a-4f48-a720-d919bdfec600 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.212127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86bc9b-6a6a-4f48-a720-d919bdfec600 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.212405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c86bc9b-6a6a-4f48-a720-d919bdfec600 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.217248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 20/41] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:39.218182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c86bc9b-6a6a-4f48-a720-d919bdfec600 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:39.231463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 22/42] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Oct 4 08:20:38 2022] POST /identity/v3/roles => generated 225 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:39.232892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.233185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.233508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.254579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:39.256252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:39.358035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c86bc9b-6a6a-4f48-a720-d919bdfec600 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:39.373084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 23/43] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:39.408627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:39.409344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.409625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.409900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.410972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:39.412917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:39.428705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:39.436600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 21/44] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Oct 4 08:20:39 2022] POST /identity/v3/roles => generated 223 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:39.666215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:39.766837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96683fa-f610-4d86-be04-61c55a87001b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:39.769065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:39.806662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:39.807297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.807579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.807855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.808960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:39.810429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd17e4-d768-4312-9060-f1a4b246d34c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:39.812040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 24/45] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:39.818369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:39.915777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314b145e-ee04-454b-a329-52911aee0a87 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:39.949619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:39.950339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:39.950687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:39.950962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:39.951650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:39.953111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c34a2f1-30e6-4114-bf23-37e4c50ed536 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:39.998577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 22/46] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:20:39 2022] POST /identity/v3/projects => generated 312 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:41.474297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69961b3e-c575-4921-a1bd-1eb208c6045b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:41.474818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69961b3e-c575-4921-a1bd-1eb208c6045b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:41.474818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69961b3e-c575-4921-a1bd-1eb208c6045b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:41.476142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 25/47] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:41.481530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:41.481980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:41.482082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:41.501092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:41.502637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:41.548026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:41.570173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 23/48] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:41.837988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69961b3e-c575-4921-a1bd-1eb208c6045b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.047246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69961b3e-c575-4921-a1bd-1eb208c6045b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.086223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.086895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.087174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.087442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.099143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-dee6125e-98a3-4066-bcb0-af1c1c5e26cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:42.104096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:42.104995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:42.105743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 26/49] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:20:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:42.111406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.181256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f9086-263c-4a99-b73b-acec4dbeae91 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.181545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f9086-263c-4a99-b73b-acec4dbeae91 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.181826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f9086-263c-4a99-b73b-acec4dbeae91 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.183236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 27/50] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:42.188407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.188695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.188974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.207413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:42.208731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:42.211283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569b478f-406e-4a40-83dd-85cbd27814aa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.237211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:42.284447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 28/51] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:42.352494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.353026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.353316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.353604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.354338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:42.354669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.356042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.365068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 24/52] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:42.370908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.470751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4635abc5-e2b7-457c-a80d-89e7fab00e3c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.506411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.507141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.507421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.507695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.508820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.510270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.511885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 29/53] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:42 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:42.517401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.560234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.562310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:42.591086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:42.622991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e04c2-c9fd-411e-93d0-fa7b824676d3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.656718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.657355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.657649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.657929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.659078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.661075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.662883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 25/54] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:42.663672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c407f-73d1-4e2e-8a4f-3fb5537f2f76 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.668777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.702201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.702867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.703150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.703424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.704559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.706024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.707932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 30/55] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:42.713436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:42.772125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145a90a-2177-4d15-9e03-a41da11005f5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.782902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:42.808208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.808830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.809113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.809387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.810130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.domain.id=default, user.id=a17a7c6f52b74569b1779d66c4e1a9d1) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:42.810437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.811935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a366419b-cb1f-4119-b0b9-59b9640deed5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.814113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:42.818258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942fa1f3-1faf-4f0f-8f45-1459c13390d0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:42.820591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 26/56] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Oct 4 08:20:42 2022] GET /identity/v3/role_assignments?user.id=a17a7c6f52b74569b1779d66c4e1a9d1&scope.domain.id=default&role.id=d7f1381a769e433aa1b033022931be96 => generated 530 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:42.857621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:42.858253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:42.858537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:42.858841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:42.859521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:42.861314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa6bbaa-d649-4bd7-80d7-def9cea499a7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:42.891422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 31/57] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Oct 4 08:20:42 2022] POST /identity/v3/projects => generated 298 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:20:45.098802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd616c-5391-4425-8c4a-841cdb2d5bc5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:45.099321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd616c-5391-4425-8c4a-841cdb2d5bc5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:45.099409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd616c-5391-4425-8c4a-841cdb2d5bc5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:45.100699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 27/58] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:45.105963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:45.106325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:45.106508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:45.125630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:45.126843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:45.193247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 32/59] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:45.464204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfd616c-5391-4425-8c4a-841cdb2d5bc5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:45.564312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfd616c-5391-4425-8c4a-841cdb2d5bc5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:45.596629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:45.597255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:45.597535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:45.597807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:45.599539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:45.601001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40cc62-98a7-4de1-bdd1-72fbca76ee1e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:45.602570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 28/60] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:45.608247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:45.685726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:45.704962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc457cfa-8c1f-4a31-b064-7cbb91488db9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:45.740034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:45.740721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:45.741071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:45.741383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:45.742083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:45.743716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872bd319-5af6-46f0-b722-5edb0d710952 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:45.769465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 33/61] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:20:45 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:20:49.236547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591739a7-ad31-400d-9329-67b620ffea85 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.237082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591739a7-ad31-400d-9329-67b620ffea85 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.237160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591739a7-ad31-400d-9329-67b620ffea85 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.238578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 29/62] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:49.252067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa79cc5-08ea-4cc1-90eb-c3aabbedb8e8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.252574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa79cc5-08ea-4cc1-90eb-c3aabbedb8e8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.252691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa79cc5-08ea-4cc1-90eb-c3aabbedb8e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.276083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-baa79cc5-08ea-4cc1-90eb-c3aabbedb8e8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:49.277556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-baa79cc5-08ea-4cc1-90eb-c3aabbedb8e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:49.364933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb90fd-5bde-403a-a082-bc784e1d5805 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.365611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb90fd-5bde-403a-a082-bc784e1d5805 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.366113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb90fd-5bde-403a-a082-bc784e1d5805 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.368366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 30/63] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:49.374976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.375547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.376012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.377201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 34/64] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:49.409349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d6f1a-01cb-4410-ba0c-067358633b2c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.410383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d6f1a-01cb-4410-ba0c-067358633b2c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.411565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519d6f1a-01cb-4410-ba0c-067358633b2c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.413615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 35/65] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:49 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:49.418648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:49.418648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:49.421822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a62f6f-7a78-4505-9b05-56dad4470fc4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.429262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a62f6f-7a78-4505-9b05-56dad4470fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.429844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a62f6f-7a78-4505-9b05-56dad4470fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.465170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-81a62f6f-7a78-4505-9b05-56dad4470fc4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:49.466517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-81a62f6f-7a78-4505-9b05-56dad4470fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:49.495013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:49.514825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 31/66] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:49.579614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:49.588916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 36/67] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:49.656176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:49.671752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50228d8b-4faf-4ec3-ae61-5945cdb1da31 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.672015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50228d8b-4faf-4ec3-ae61-5945cdb1da31 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.672229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50228d8b-4faf-4ec3-ae61-5945cdb1da31 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.674322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 37/68] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:49.685496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377a5c3-eddd-4f00-a610-7c03b6021019 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.685770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377a5c3-eddd-4f00-a610-7c03b6021019 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.686056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377a5c3-eddd-4f00-a610-7c03b6021019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.727841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b377a5c3-eddd-4f00-a610-7c03b6021019 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:49.729292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b377a5c3-eddd-4f00-a610-7c03b6021019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:49.777303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f721c-ac0a-4836-80ca-84eee9e6d855 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:49.815416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:49.816450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.816920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.817965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.828986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-dda945fe-b5d8-486e-94a1-31b025884e49 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:49.835613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:49.836472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:49.837196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:49.844200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 32/69] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:49.844481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e6154-5473-4645-979d-77d9ca5ffdb0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.844952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e6154-5473-4645-979d-77d9ca5ffdb0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.845067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e6154-5473-4645-979d-77d9ca5ffdb0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.846447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 33/70] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:49.847043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e6154-5473-4645-979d-77d9ca5ffdb0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:49.868096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 38/71] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:49.870654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:49.871207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:49.871872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:49.885017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:49.894904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:49.896970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:49.959338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e6154-5473-4645-979d-77d9ca5ffdb0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:49.984468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 39/72] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:49.985111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.000403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.001296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.001765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.002223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.003126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:50.003635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.005991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.019784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 34/73] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:20:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.021712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.059492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.122832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e290ad-0c65-4f67-907f-836e3a464642 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.131617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.139981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ada7-fe9a-4d43-8bca-0eb657f95f7d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.177832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.178643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.179244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.179520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.186320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.186990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.187277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.187557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.191228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:50.197628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:50.198453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:50.199263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:50.199834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:50.202124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 40/74] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:20:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:50.202124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:50.203471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:50.204308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:50.205031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 35/75] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:50.205650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.309150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1573e15-5300-4a7a-8cbb-b825395f276d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.310662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af59af-af42-4a6e-8db1-8a2e27f0c3db None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.325732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.343982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.344617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.344898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.344987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.345173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.345610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.345889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.346162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.355825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:50.359274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:50.360647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:50.361885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:50.363279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:50.364736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.364811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 36/76] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:50.364956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 41/77] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.365023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.368347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.379572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.471753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeedf1b-29b2-48b0-b38b-731f5c6bd966 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.472388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805ef45-0e76-4da2-b19b-423588653a37 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.507149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.507752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.508030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.508117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.508303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.508672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.508952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.509033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:50.509225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.509316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.509885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:50.510189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.510637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.511596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.518535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 42/78] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.519337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.525359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 37/79] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.525359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.561988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.602266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.626415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b38b534-ed2f-479c-b684-3033b82091ad None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.628621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca3c485-79c0-4807-a56b-8378e38501f0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.661020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.661744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.661996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.662121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.662271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.662726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.662978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:50.663074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.663262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.663344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.664375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.664653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.665809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.667410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 43/80] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.668197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.673351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 38/81] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.674214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.778874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35962900-a3ee-45a2-afcb-864f90bbc27e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.781128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b73bd2-5787-4901-8be6-89e565552cd4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.781267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.814850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.815390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.815672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.815952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.817326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.818082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.818371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.818687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.819941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.821471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.823142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 39/82] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.823893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.833283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:50.889262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.890792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.931478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6289f2-2489-4886-88cd-9bad6f680277 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:50.966576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:50.967240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:50.967522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:50.967799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:50.968953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:50.970393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:50.972018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 40/83] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:50.972764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:50.996414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 44/84] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:20:50.997276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.000389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.052400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.081101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c592a92-b050-4212-842d-957350c4b358 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.104718 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b0809-99da-440c-8209-52f97838c61b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.117412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.118022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.118306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.118604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.119274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.120744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.139732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.140371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.140749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.140924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.143301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.145257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.147423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 45/85] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:50 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:51.148278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.159476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 41/86] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:20:50 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:51.159476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.234989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.270223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193b8e1-cce1-4960-94f1-59102e2e039b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.280408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251250da-40b1-42d9-9264-232e33d98495 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.310849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.311520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.311746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.312043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.320352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.330171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.330171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.330171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.330171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.392936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.394418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.419189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.420691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.483433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.525044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 46/87] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:20:51.527089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.541477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 42/88] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:20:51.542392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.577378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.649480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e46a414-e99e-4dde-a9e8-15bcca638ae4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.675667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b01045-8cd5-4d6f-adeb-3bf1958eb494 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.689066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.689911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.690326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.690874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.692366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.694108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.695986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 47/89] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:51.701949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.723291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.724510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.724792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.725063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.760828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.805515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.807369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.839528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:51.840580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8c4595-a7a4-4372-9a80-9aa04189be41 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:51.896533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:51.897174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:51.897450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:51.897719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:51.899272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:51.901653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:51.903296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 48/90] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:51.904158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:51.933271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 43/91] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:20:51.934501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:52.037362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a659c-cc68-4946-bde8-35ca5821d9ce None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.045498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a3122-bfd9-4172-994c-59b685884cf6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.050759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:52.085583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.086695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.087223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.087512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.088678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.088935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:52.091059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.094602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 49/92] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:52.094602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:52.102782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.103425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.103706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.103977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.104710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:52.105016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.106450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.119039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 44/93] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:20:51 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:52.120120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:52.254270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de84e007-d8bc-4543-8fa5-f8588b3795cf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.300988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.301659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.301994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.302329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] PATH_INFO: `/v3/projects/6667c66651ee4e27bebed7dcf8842a63` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.306808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] RBAC: Authorizing `identity:get_project(project_id=6667c66651ee4e27bebed7dcf8842a63)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.308293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.315201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 45/94] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:52 2022] GET /identity/v3/projects/6667c66651ee4e27bebed7dcf8842a63 => generated 312 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:52.316072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:52.351946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a04ef98-a8e0-411f-9b60-c258d8bb8d0c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.401700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.402311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.402610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.402882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.403611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:52.403918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.405379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.421426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 50/95] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:20:52 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 259 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:52.422009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:52.468677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:52.514006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:52.541170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875a6627-ab97-4295-be9a-4519ecad4799 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.579894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.580559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.581026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.581369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.582179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=a17a7c6f52b74569b1779d66c4e1a9d1) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:52.582670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.586578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79c9909-5d2d-44ea-9a8d-9e7a9505f80f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.591836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d575be-6e92-458e-81c4-a710d432df5b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:52.597615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 46/96] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:20:52 2022] GET /identity/v3/role_assignments?user.id=a17a7c6f52b74569b1779d66c4e1a9d1&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:52.635027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:52.635662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:52.635944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:52.636217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:52.636975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=6667c66651ee4e27bebed7dcf8842a63, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:52.637284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:52.638849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6178f3e-33cf-490b-8be0-a0ee986aa423 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:52.646183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 51/97] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:20:52 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=6667c66651ee4e27bebed7dcf8842a63&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:53.856839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814c4b9-d632-491b-9fea-4eb31ec8273d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:53.857433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814c4b9-d632-491b-9fea-4eb31ec8273d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:53.857433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814c4b9-d632-491b-9fea-4eb31ec8273d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:53.859119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 47/98] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:53.865467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:53.866363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:53.866894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:53.889063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:53.890799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:53.928520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3392e-70d6-4463-a186-d9206ee03449 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:53.928879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3392e-70d6-4463-a186-d9206ee03449 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:53.929093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c3392e-70d6-4463-a186-d9206ee03449 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:53.930755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 48/99] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:53.937523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:53.937807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:53.938084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:53.968995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:53.970205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:54.033880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 52/100] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:54.105010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 49/101] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:54.429032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:54.473023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:54.545649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f397e-4fc4-48fe-a8b7-3a3bfc7a2877 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:54.581110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:54.581739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:54.582017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:54.582290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:54.583995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:54.585440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:54.587022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 53/102] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:54.595583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:54.664261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:54.688066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:54.728746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c516e66e-0af0-4012-a136-f7707d135025 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:54.801336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:54.802786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:54.803069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:54.803342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:54.804000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:54.805440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:54.864246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 50/103] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:20:54 2022] POST /identity/v3/projects => generated 302 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:54.897959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc42706-ac63-42d3-aada-178a26ad8b58 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:54.901862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:54.946664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:54.947234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:54.947516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:54.947793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:54.963196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:54.968165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:54.969017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:54.969762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 54/104] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:54.975556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:55.139841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e7a931-3e2e-4011-a1bc-f1513b00e1e3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:55.153513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:55.190121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:55.190789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:55.191074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:55.191349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:55.192028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:55.192334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:55.193991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:55.206435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 51/105] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:20:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:55.213066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:55.358065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac07dc-e09c-4e81-a629-b55db9e74ae2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:55.410131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:55.410800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:55.411088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:55.412035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] PATH_INFO: `/v3/users/c0c72f859558413bb33abbd3c8f0235b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:55.430293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] RBAC: Authorizing `identity:get_user(user_id=c0c72f859558413bb33abbd3c8f0235b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:55.431818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7864897-129f-48a1-ae0d-201f9411ee6c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:55.434846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 55/106] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:55 2022] GET /identity/v3/users/c0c72f859558413bb33abbd3c8f0235b => generated 289 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:55.439564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:55.592733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec547b0-333e-4f23-8f22-11ff06b423ad None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:55.631556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f81192-61b7-4362-90aa-4e0f63604970 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:55.631869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f81192-61b7-4362-90aa-4e0f63604970 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:55.632109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f81192-61b7-4362-90aa-4e0f63604970 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:55.633459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 56/107] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:55.637434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:55.638078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:55.638363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:55.638663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:55.638980 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:55.639361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:55.639727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:55.663851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:55.665059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:55.690822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:55.709619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:55.761066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:55.762529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9809681-c04d-47ca-8be0-af7fcfeff3c6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:55.796700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 57/108] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:55.921064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 52/109] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:20:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:20:55.927903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.055835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0d371c-116c-42a5-b511-eadb29953291 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.056165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0d371c-116c-42a5-b511-eadb29953291 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.056385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0d371c-116c-42a5-b511-eadb29953291 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.057863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 53/110] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:20:56.065321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841483ce-65af-485c-90de-4717535f984b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.065601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841483ce-65af-485c-90de-4717535f984b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.065927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841483ce-65af-485c-90de-4717535f984b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.075063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65368c-6e48-4398-93bc-1749f17fe3e2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.100415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-841483ce-65af-485c-90de-4717535f984b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:56.101663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-841483ce-65af-485c-90de-4717535f984b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:56.152094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.152739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.153079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.153350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.154575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:56.157159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8fbb4b-dd96-4db3-a6d4-5b81ed2b8e5b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:56.159043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 58/111] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:55 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:56.159962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfb957-4279-408f-82b2-f37e83fb777c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.160239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfb957-4279-408f-82b2-f37e83fb777c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.160578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cfb957-4279-408f-82b2-f37e83fb777c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.164199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 59/112] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:56.170329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.170643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.170917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.193122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:56.194756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:56.208526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.210580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.219013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 54/113] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:20:56.221385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841483ce-65af-485c-90de-4717535f984b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.293723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 60/114] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:56.294883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.342959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841483ce-65af-485c-90de-4717535f984b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.384776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.385396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.385675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.385950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.386717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=c0c72f859558413bb33abbd3c8f0235b) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:56.387029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:56.388591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f89a8a6-910c-45ec-bf5b-d9faaa0ab613 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:56.396296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 55/115] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/role_assignments?user.id=c0c72f859558413bb33abbd3c8f0235b&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:56.398266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d29e-1065-4ded-aff9-dd8a5c444d93 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.398716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d29e-1065-4ded-aff9-dd8a5c444d93 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.398999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d29e-1065-4ded-aff9-dd8a5c444d93 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.400451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 56/116] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:56.404977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13db30f-c44f-46ed-8df7-24a8e215b5e2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.407917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.410571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.410571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.451081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:56.455451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:56.493930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.494651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.494937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.495211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.523951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:56.530598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:56.531426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:20:56.532154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:56.533674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 61/117] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:56.539702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.563558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.594958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.608122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 57/118] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:56.609338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.737151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc21e65-76bf-4b35-ad8a-bc4f013d0768 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.760558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b56adb9-9af5-49b8-930a-b9bcb0fe8367 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.779987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.780874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.781320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.781757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.782649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:56.783119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:56.784663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:56.793797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 62/119] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:20:56.794957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.804162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.804806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.805086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.805357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.819768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:56.826888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:56.827724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:20:56.828461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 58/120] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:56.829071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:56.902235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.949440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:56.951982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaa71db-eb40-4952-8c1e-1cb9775ecd80 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:56.990713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:56.992183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:56.992637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:56.993070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:56.994539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:56.996451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:56.998323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 59/121] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:56.999686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.014115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4f580c-be86-4fe4-8b5b-34bd761219f8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.057066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.057955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.058383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.058831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.074228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:57.082017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:57.083075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:20:57.084455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:57.086774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 63/122] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:57.088223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.115312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f825b5a2-3030-4fa5-b535-511c887d35f9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.148025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.164590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.165408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.165835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.166275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.167110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:57.167555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.169130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.184442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 60/123] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:20:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.185628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.226514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957e00f6-e739-4bee-9669-4343fe397a6f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.263523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.264526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.265005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.265494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.266461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.279765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:20:57.285299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:20:57.286149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:20:57.287529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:20:57.289854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 64/124] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:20:57.290988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.311232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7e245-258b-4d23-a329-0ec73375c02a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.350094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.351201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.351680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.352145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.353489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.355326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.357175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 61/125] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.358152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.412551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a4584-ee9c-4ee8-a1f3-c5447c7bf776 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.433219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.454188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.455406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.455889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.456358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.457230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:57.457753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.460374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.469620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 65/126] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.470953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.500260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa306a5-ab50-478b-9572-6fab10794627 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.531840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.549521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.550232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.550525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.550830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.552394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.554793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.556398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 62/127] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.558591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.591495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbcde75-33a8-4268-aa3e-26dbeb357514 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.626690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.627455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.627803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.628179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.628908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:20:57.629284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.631348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.640337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 66/128] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.641084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.704191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c25cb-5a72-4992-a657-a5f446384cba None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.732124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.748128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031b7e7d-560f-4941-bd1c-281a3ff6d275 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.750870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.751579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.754704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.755046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/829a99ce94f043fb93bddd51c7961fbe/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.758863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, user_id=829a99ce94f043fb93bddd51c7961fbe, role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.760789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.763728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:57.778336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 63/129] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:20:57 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/829a99ce94f043fb93bddd51c7961fbe/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:20:57.779344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.789404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.790428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.790937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.791554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.793076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.794901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.797191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 67/130] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.798354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.918158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5ee7b0-ae45-4246-9b31-abc769163dca None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.922321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9783132-cd7e-4700-9d2d-b5bff8b8c760 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:57.965292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:57.966043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:57.966232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:57.966511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:57.967743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:57.969581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:57.971560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 64/131] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:57.972377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:57.999342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.000807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.001287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.001869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.007059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.009274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.011743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 68/132] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:58.012852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:58.039915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:58.085069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:58.107758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ede20-300e-44c4-980a-cdbc1135cb11 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:58.147571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f65da1-94f9-429a-8926-2dde0e9e8860 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:58.162232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.163114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.163401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.163686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] PATH_INFO: `/v3/projects/6667c66651ee4e27bebed7dcf8842a63` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.165679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] RBAC: Authorizing `identity:get_project(project_id=6667c66651ee4e27bebed7dcf8842a63)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.167729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.169885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 65/133] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:57 2022] GET /identity/v3/projects/6667c66651ee4e27bebed7dcf8842a63 => generated 312 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:58.170777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:58.192005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.193387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.193740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.194085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/829a99ce94f043fb93bddd51c7961fbe/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.196143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, user_id=829a99ce94f043fb93bddd51c7961fbe, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.197669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5711effb-b300-47ce-8bbc-813156783c5b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.216182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 69/134] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:20:58 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/829a99ce94f043fb93bddd51c7961fbe/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:20:58.217635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd11188-5e16-45fb-9a33-bf1453336b59 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.218019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd11188-5e16-45fb-9a33-bf1453336b59 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.218358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd11188-5e16-45fb-9a33-bf1453336b59 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.219934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 70/135] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:58.220769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd11188-5e16-45fb-9a33-bf1453336b59 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:58.357462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd11188-5e16-45fb-9a33-bf1453336b59 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:58.380833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:58.405344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6c107b-06ca-4207-9855-427131aa32a6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:58.420495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.421244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.421530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.421802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] PATH_INFO: `/v3/projects/6667c66651ee4e27bebed7dcf8842a63/users/829a99ce94f043fb93bddd51c7961fbe/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.428660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] RBAC: Authorizing `identity:create_grant(project_id=6667c66651ee4e27bebed7dcf8842a63, user_id=829a99ce94f043fb93bddd51c7961fbe, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.430174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8d3c65-8237-4d15-9d06-447e551cb3fc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.438931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:58.457444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 71/136] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:20:58 2022] PUT /identity/v3/projects/6667c66651ee4e27bebed7dcf8842a63/users/829a99ce94f043fb93bddd51c7961fbe/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:20:58.458906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.459184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.459460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.472289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.473362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.473855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.474990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.478081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.480914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.484182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 66/137] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:20:58 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:58.486279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:58.487890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:58.494766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:58.569293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 72/138] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:58.691000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4a0cc8-7fda-427f-a15e-6806809392ff None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:58.733089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:58.733779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:58.734174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:58.734482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/a17a7c6f52b74569b1779d66c4e1a9d1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:58.739344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, user_id=a17a7c6f52b74569b1779d66c4e1a9d1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:58.741205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:58.756581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 67/139] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:20:58 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/a17a7c6f52b74569b1779d66c4e1a9d1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:20:58.933255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:59.047995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:59.050953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c62e0a-111b-45a6-a90a-301d332a4ea3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:59.092858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:59.093516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:59.093792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:59.094058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:59.095445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:59.096944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5127ae2f-e893-4bed-b914-37140f1cad1a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:59.099049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 73/140] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:20:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:20:59.104627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:20:59.159012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:20:59.225490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a10c0f-4c0b-4415-b6d9-f64e995d0072 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:20:59.262344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:20:59.262998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:59.263280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:59.263548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:59.264223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:20:59.265659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37095e27-3725-469f-92ba-e47d629f253a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:20:59.290966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 68/141] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:20:59 2022] POST /identity/v3/users => generated 283 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:20:59.385630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12ef05-84f7-462c-8f3c-cf26eae1c443 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:59.386010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12ef05-84f7-462c-8f3c-cf26eae1c443 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:59.386189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12ef05-84f7-462c-8f3c-cf26eae1c443 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:59.387754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 74/142] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:20:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:20:59.395243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:20:59.395551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:20:59.395788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:20:59.423682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:20:59.424895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:20:59.517910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 69/143] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:20:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:20:59.949568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12ef05-84f7-462c-8f3c-cf26eae1c443 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:00.078847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12ef05-84f7-462c-8f3c-cf26eae1c443 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:00.126697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:00.127691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:00.128173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:00.128642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:00.143662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:00.149527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:00.150425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:00.151217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:00.153735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 75/144] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:20:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:00.160149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:00.198135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:00.292649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d55675-3efc-4914-aa78-5b7fcbc92aa2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:00.342678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:00.343294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:00.343571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:00.343838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:00.344572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:00.344876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:00.346256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:00.358296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 70/145] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:00.364487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:00.396581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:00.479776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a41b8-aa13-4147-8839-73368dfb1a1b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:00.525145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:00.525782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:00.526065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:00.526392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] PATH_INFO: `/v3/users/c0c72f859558413bb33abbd3c8f0235b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:00.527614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] RBAC: Authorizing `identity:get_user(user_id=c0c72f859558413bb33abbd3c8f0235b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:00.529117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:00.530892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 76/146] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:00 2022] GET /identity/v3/users/c0c72f859558413bb33abbd3c8f0235b => generated 289 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:00.536506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:00.650938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:00.660431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e042ac7c-9482-4ebd-9510-743e2001a116 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:00.700284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:00.700932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:00.701207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:00.701476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:00.702656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:00.704122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32306fb4-e911-4545-9fa4-4f112e482881 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:00.705855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 71/147] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:00 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:00.711957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:00.841890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:00.845568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f01261-80bc-47a9-9569-74f9de2d3ccd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:00.890348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:00.891047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:00.891324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:00.891593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/c0c72f859558413bb33abbd3c8f0235b/roles/b1f43cce7d8c45a6bea928386ae2bb04` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:00.893525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, user_id=c0c72f859558413bb33abbd3c8f0235b, role_id=b1f43cce7d8c45a6bea928386ae2bb04)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:00.895024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7500f07c-8410-49e6-a2e4-233de5ad7960 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:00.912181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 77/148] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:21:00 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/users/c0c72f859558413bb33abbd3c8f0235b/roles/b1f43cce7d8c45a6bea928386ae2bb04 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:01.411022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1dd79-fbfa-4003-bcb2-15d835d9febb None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.412706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1dd79-fbfa-4003-bcb2-15d835d9febb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.413207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1dd79-fbfa-4003-bcb2-15d835d9febb None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.414863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 72/149] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:01.420840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85865fd-f721-461c-b59f-42eeed0e18fb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.421800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85865fd-f721-461c-b59f-42eeed0e18fb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.422285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85865fd-f721-461c-b59f-42eeed0e18fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.446186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a85865fd-f721-461c-b59f-42eeed0e18fb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:01.447781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a85865fd-f721-461c-b59f-42eeed0e18fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:01.531537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 78/150] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:01.857541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61e11e-bee4-47e8-87ab-c0806063cc3b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.858199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61e11e-bee4-47e8-87ab-c0806063cc3b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.858720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e61e11e-bee4-47e8-87ab-c0806063cc3b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.860386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 73/151] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:01.866957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.867587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.868068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.891275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:01.892815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:01.935886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f2cbe4-3075-4d01-82c1-004f5919e63e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.936539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f2cbe4-3075-4d01-82c1-004f5919e63e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.937018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f2cbe4-3075-4d01-82c1-004f5919e63e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.938856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 74/152] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:01.944275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:01.944867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:01.945378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:01.975754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:01.978225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:02.014272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 79/153] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:02.078354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 75/154] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:02.184179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:02.209077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:02.316206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2f371-6c04-4dcb-9819-22b60cec25e9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:02.363923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:02.365001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:02.365640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:02.366125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:02.381321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:02.386754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:02.387675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:02.388457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:02.390765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 80/155] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:02.397599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:02.499208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:02.517511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b5ab53-f506-46ca-9a59-74f3788ce8bf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:02.565417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:02.566458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:02.566969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:02.567458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:02.568710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:02.569234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:02.570875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:02.587541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 76/156] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:02.593405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:02.675596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:02.711998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:02.760701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132f9da4-2507-46bc-813f-1561e92745a6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:02.871441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:02.872240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321852fd-817b-4096-b530-62fa61ecb01e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:02.872741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:02.873224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:02.873700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:02.879404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:02.881335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:02.883635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 81/157] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:02.884739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:02.951294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:02.951934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:02.952212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:02.952479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:02.973858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:02.980313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:02.981303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:02.982061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 77/158] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:02.992649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:03.016778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:03.096157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a4eb0-5d28-460b-b4a0-e3addb0901c6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:03.182314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a801764-4590-4b21-a52f-fdc1e705883a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:03.190024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:03.190695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.190974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.191239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.207279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2f5472ed-c76c-4294-bc98-c118dc1dfc54 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:03.220890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:03.221815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:03.222693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 82/159] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:03.223335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da773bbf-fe1e-43f4-ab6f-7fa73ac3a661 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.224152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da773bbf-fe1e-43f4-ab6f-7fa73ac3a661 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.225555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da773bbf-fe1e-43f4-ab6f-7fa73ac3a661 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.230739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 83/160] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:03.231680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da773bbf-fe1e-43f4-ab6f-7fa73ac3a661 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:03.239740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:03.264590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:03.265317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.265593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.265859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.395846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da773bbf-fe1e-43f4-ab6f-7fa73ac3a661 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:03.442104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:03.444423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9c3fd-02e5-4045-a255-6465bcfb58d0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:03.468242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:03.468872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.469147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.469417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.470096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:03.470403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:03.471833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:03.473794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:03.490856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 84/161] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:03.491750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:03.572759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:03.626680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fbdb9b-2a49-4f11-8bb2-977c4fe376ae None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:03.702962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:03.703519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.703795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.704310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 78/162] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 720 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:03.705648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.705922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.706186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.708659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.709398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:03.709756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:03.711176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:03.730291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 85/163] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:03.732777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:03.748188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:03.749380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:03.876106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de17499-5363-4286-a7aa-f66fa790d152 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:03.907163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:03.916428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 79/164] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:03.917309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:03.917937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:03.918826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:03.919257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:03.919665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:03.920987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:03.922696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:03.924857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 86/165] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:03 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:03.925863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:04.011074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:04.036686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628b2c32-ed07-4828-8e1c-7628daa55691 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:04.088154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:04.089104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:04.089541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:04.089951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] PATH_INFO: `/v3/users/829a99ce94f043fb93bddd51c7961fbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:04.091367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] RBAC: Authorizing `identity:get_user(user_id=829a99ce94f043fb93bddd51c7961fbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:04.092998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:04.094755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 87/166] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:03 2022] GET /identity/v3/users/829a99ce94f043fb93bddd51c7961fbe => generated 275 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:04.095773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:04.105243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574fca3-81fd-4da0-a3f0-e497f87ca4c6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:04.167751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:04.187450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:04.188080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:04.188412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:04.188790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] PATH_INFO: `/v3/projects/6667c66651ee4e27bebed7dcf8842a63` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:04.190092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] RBAC: Authorizing `identity:get_project(project_id=6667c66651ee4e27bebed7dcf8842a63)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:04.191593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694bc8d2-2a2c-4169-9175-548e8b171c60 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:04.193500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 80/167] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:03 2022] GET /identity/v3/projects/6667c66651ee4e27bebed7dcf8842a63 => generated 312 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:04.207596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac636d2-3cba-47d6-8d66-f6eef2b8d4d6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:04.208047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac636d2-3cba-47d6-8d66-f6eef2b8d4d6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:04.209315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac636d2-3cba-47d6-8d66-f6eef2b8d4d6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:04.210633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 81/168] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:04 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:04.211413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac636d2-3cba-47d6-8d66-f6eef2b8d4d6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:04.221021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0f129-45ed-4ce7-a1e0-f4ddceae6fea None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:04.261790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:04.263486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:04.263958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:04.264361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:04.410575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:04.412928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:04.524529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac636d2-3cba-47d6-8d66-f6eef2b8d4d6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:04.553101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:04.609707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:04.662581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:04.662581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:04.662581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:04.662581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:04.716259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 88/169] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 621 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:04.717337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:04.900933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:04.902596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df807a2-6815-486f-840f-9904425aa91b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:04.997773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f0580-d132-4874-b819-fc95f8920677 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.022585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 82/170] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 811 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:05.024275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.024697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.025058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.041875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.042820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.043286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.043731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.045120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=6667c66651ee4e27bebed7dcf8842a63, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:05.045819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:05.046606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:05.047951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:05.048540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:05.051813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.065162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 89/171] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:04 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=6667c66651ee4e27bebed7dcf8842a63&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 608 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:05.067080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.104037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.140009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 83/172] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:05.141231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.218198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34bfb17-9d50-4bc9-ad87-eb5e2c3d3990 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.269648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1a4edb-769e-40a1-91fa-2f0fe6918e48 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.288228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.289895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.290503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.291570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.309573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fb3b534e-3057-4b30-b61d-85e59b73b2a1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:05.320038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:05.321617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:05.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 90/173] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:05.324513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e008c76-6b04-453c-8c8a-601f84be7d91 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.324513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e008c76-6b04-453c-8c8a-601f84be7d91 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.325584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e008c76-6b04-453c-8c8a-601f84be7d91 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.330760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 91/174] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:05.332212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e008c76-6b04-453c-8c8a-601f84be7d91 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.332598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.334474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.336215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.336973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.340775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:05.342438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ade8688-cadd-46ce-8add-df89dfb41e4b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:05.344325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 84/175] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:05.345674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431de30-4561-49db-8f63-07fd070d6908 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.346170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431de30-4561-49db-8f63-07fd070d6908 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.346540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431de30-4561-49db-8f63-07fd070d6908 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.347921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 85/176] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:05.349074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a29dff-cf64-4714-af95-0efc736d06ac None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.350534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a29dff-cf64-4714-af95-0efc736d06ac None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.351094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a29dff-cf64-4714-af95-0efc736d06ac None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.352546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 86/177] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:05.353679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.354759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.356296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.377441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:05.378657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:05.380734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.456544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 87/178] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:05.457008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.457634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.474197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e008c76-6b04-453c-8c8a-601f84be7d91 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.509680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.510302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.510600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.510874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.511978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:05.513458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8ade04-1766-417d-8405-e984a54e631e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:05.519723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 92/179] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:05.519723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.519723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.519723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.538337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:05.539570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:05.580699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7033cb46-22ce-4a87-a589-9378f16a7975 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.625957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 93/180] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:05.628450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.629063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.629329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.629598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.630449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:05.630909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:05.632424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4dd8f0-fcf5-4583-8cf2-389cf50e68a4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:05.636063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.638774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.652795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 88/181] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:05.654775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.655054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.655438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.675283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:05.676461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:05.752215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.768499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 89/182] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:05.769508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.784111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed83a05c-37e7-4882-8dfb-ee792d79df79 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.824261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.825393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.825835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.826740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.827921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:05.828633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:05.830659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:05.839078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 94/183] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 608 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:05.840845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:05.895450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be47381-8cca-4a52-a3a9-8ec438eec680 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:05.934741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:05.943375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:05.944622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:05.945080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:05.945494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:05.958647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:05.964313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:05.965428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:05.966417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:05.968865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 90/184] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:05.969877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.007961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a543e2-2729-4fa1-9fe8-ead81f26ff03 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.063059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.063668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.063944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.064212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.064943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=829a99ce94f043fb93bddd51c7961fbe) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:06.065243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.066742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.074266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 95/185] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/role_assignments?user.id=829a99ce94f043fb93bddd51c7961fbe&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.075220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.095282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.128482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b3d3b9-24d6-4e12-8c5a-f3a2d907de0e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.202697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.203476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.203761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.204030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] PATH_INFO: `/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.206977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] RBAC: Authorizing `identity:get_user(user_id=a17a7c6f52b74569b1779d66c4e1a9d1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.209718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.211538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 91/186] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:05 2022] GET /identity/v3/users/a17a7c6f52b74569b1779d66c4e1a9d1 => generated 247 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.212658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.221520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a8cff-861c-4a7e-b234-69ac11705837 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.269106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.269639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.270863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.270863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.284455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:06.289948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:06.291268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 96/187] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:06.293891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.295495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.360669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f408c4cc-b139-4f42-94cd-06a372f7b57d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.403048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.407574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.408204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.408476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.408738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.422420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:06.427196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:06.428196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 92/188] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:06.429219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.431125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e955ca7-75ea-4bf5-9e31-af1b55a3aa75 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.467917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.468530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.468797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.469065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.469731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:06.470029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.471456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.480551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 97/189] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.481391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.531197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce18e686-6c5d-4097-b430-71fcdaf9f0c3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.547763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.573948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.574595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.574899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.575333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.576453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.577906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.579496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 93/190] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.580262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.585037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772f6aa5-53e1-4d7b-bebc-e88dd4f4eebf None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.619085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.619686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.619963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.620234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.623595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.686427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdfeaf9-ca0d-4300-bf78-bd14509c5318 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.689454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.690955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.720942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.721663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.722035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.722422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.723238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:06.723645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.726477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.736159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 94/191] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.737013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.796025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.801609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 98/192] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:06.802568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.839542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:06.854629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d15941-6e6d-4a91-ad09-67e2d05af4f8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.891659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.892268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.892544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.892817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.893480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:06.893799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.895209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.904336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 95/193] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.904336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:06.914848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1514374-39ff-4a61-8c93-7396cd89f7e3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:06.952066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:06.952700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:06.952977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:06.953242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:06.967434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:06.969239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:06.970918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 99/194] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:06.971692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.013903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b54e43-b81d-4c9b-ab37-3d9f6b2d02a2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.025132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.048236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.048757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.049028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.049299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.049938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.051417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.065627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.073546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 96/195] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:06 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:07.074338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837225b-65e5-4682-b2e9-f3ab14d0d1b8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.074442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.115803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.116875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.117355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.117804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.119297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.121149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.123631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 100/196] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:06 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:07.124993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.223074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6c8a41-6dbf-44fd-b9d4-936fdff279dc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.252797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f229-1cbf-4307-a954-6c38a3643763 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.267651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.268297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.268581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.268844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.270372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.271544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.273663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 97/197] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:07.274505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.293955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.294377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.294686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.295901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] PATH_INFO: `/v3/users/c0c72f859558413bb33abbd3c8f0235b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.297068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] RBAC: Authorizing `identity:get_user(user_id=c0c72f859558413bb33abbd3c8f0235b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.298627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.301486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 101/198] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/users/c0c72f859558413bb33abbd3c8f0235b => generated 289 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:07.302148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.329684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.340917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.437612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c294838a-0982-4de4-92a8-34b5d3e2de96 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.451577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff86ec-5d11-4067-8736-173bf3f4d35a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.487197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.488077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.488405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.488669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.496384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.497010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.497294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.497568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.570597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.572074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.572971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.574622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.618357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.621706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.716369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 98/199] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:07.717843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.721654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 102/200] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:07.723720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.847658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ece3f-8f18-429e-a8f2-bedc7b598732 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.859057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c7a32-9719-4720-8e8e-94c10e0c73e5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:07.893369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.894286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.894824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.895191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.895905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.896111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=a17a7c6f52b74569b1779d66c4e1a9d1) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:07.896584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.898388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:07.900510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:07.906065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 99/201] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/role_assignments?user.id=a17a7c6f52b74569b1779d66c4e1a9d1&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:07.907034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:07.908390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:07.909227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:07.909656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:07.910201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:07.997786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:07.999637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.039926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55745580-1a8f-46a6-959c-7e34f85207aa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.097185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.098354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.098958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.099424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.101602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.103397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.106497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 100/202] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:08.107689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:08.167628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 103/203] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:08.168630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:08.194981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:08.208729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:08.238578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34af1001-e50b-49d4-84fa-db02dbdc1db5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.280931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.282352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.282899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.283349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.284981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.289264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.292209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 101/204] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:08 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:08.294114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:08.301586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6025bc6-7594-4130-97ef-89882f7032bd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.353553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.354323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.354642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.355012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.355894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=4a556d2b935246128748703dc8cf8002) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:08.356304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.357879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.364242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 104/205] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:08 2022] GET /identity/v3/role_assignments?user.id=4a556d2b935246128748703dc8cf8002&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:08.365002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:08.434282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a60817-d0c0-40d0-a767-59a56cae7a50 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.482583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.482583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.482583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.482583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.485137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.486625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.489990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 102/206] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:08 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:08.495469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:08.501846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:08.507807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:08.515541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66377bcb-7008-439e-88cd-1346dec06319 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.563485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.564285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.564677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.565041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.565891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=4a556d2b935246128748703dc8cf8002) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:08.566304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.568032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3e756-dc93-4bca-811a-ae5b8660007d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.575409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 105/207] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:08 2022] GET /identity/v3/role_assignments?user.id=4a556d2b935246128748703dc8cf8002&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 259 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:08.696400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b04d2d3-cf85-45f9-9abb-a87f4c36da3b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:08.750286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:08.750954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:08.751229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:08.751497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:08.752246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d, user.id=c0c72f859558413bb33abbd3c8f0235b) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:08.752545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:08.754047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb497af-69fc-45c6-ab5e-aa14a932d2a2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:08.763636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 103/208] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:08 2022] GET /identity/v3/role_assignments?user.id=c0c72f859558413bb33abbd3c8f0235b&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 608 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:09.748209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ebe406-1038-4f2b-bc49-ef680dd31f0d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:09.748758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ebe406-1038-4f2b-bc49-ef680dd31f0d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:09.748758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ebe406-1038-4f2b-bc49-ef680dd31f0d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:09.751291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 106/209] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:09.759497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:09.759992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:09.760069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:09.778727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:09.779930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:09.845485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017ee65-f58a-4dc3-9f98-344b0c9938a3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:09.845821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017ee65-f58a-4dc3-9f98-344b0c9938a3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:09.846705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0017ee65-f58a-4dc3-9f98-344b0c9938a3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:09.847772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 107/210] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:09.851347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:09.853677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:09.853954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:09.854220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:09.865143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 104/211] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:09.876528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:09.877710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:09.951602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 108/212] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:10.139866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.232185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.249445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef50d23-19c8-46da-a28f-62f085c29779 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.286599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.287891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.288343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.288847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.299966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:10.305471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:10.306353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:10.307189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 105/213] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:10.312583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.338799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9214039f-706e-486b-acc9-33bf8a9d9576 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.342992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:10.376999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:10.378691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.379507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.379955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.380234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.382493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:10.384272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:10.390943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 109/214] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:10.391854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.429558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bd787-7a05-4310-aa35-a53773f89117 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.467908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.468912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.469389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.469855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.470770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:10.471301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:10.473011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:10.483053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 106/215] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:10.489642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.503929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a22799d-0e3e-4146-8fcd-566d44eb13cf None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.542834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.543593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.543771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.544026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.544707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:10.546187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:10.578972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 110/216] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:10 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:21:10.612091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267e1fe-a6ae-4cad-98d0-74a144133fb7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.634567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:10.656498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.657077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.657352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.657622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] PATH_INFO: `/v3/users/11952b25a37d473b9c5591cced5dd912` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.676969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] RBAC: Authorizing `identity:get_user(user_id=11952b25a37d473b9c5591cced5dd912)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:10.679405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821b088c-655a-41c7-ba7e-561b43740c52 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:10.681216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 107/217] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:10 2022] GET /identity/v3/users/11952b25a37d473b9c5591cced5dd912 => generated 297 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:10.688752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:10.707339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:10.828008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92a327-654f-4f28-9d09-2696ded23754 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:10.867895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:10.868605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.868883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.869158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.925579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf53107-40b5-4f21-a2b6-493464704546 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.926049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf53107-40b5-4f21-a2b6-493464704546 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.926049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf53107-40b5-4f21-a2b6-493464704546 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.927414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 108/218] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:10.934389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58ac24-9d1b-47a7-b0dd-98d7766bdc1c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:10.934863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58ac24-9d1b-47a7-b0dd-98d7766bdc1c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:10.935137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e58ac24-9d1b-47a7-b0dd-98d7766bdc1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:10.955173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:10.957265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:10.960990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4e58ac24-9d1b-47a7-b0dd-98d7766bdc1c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:10.962202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4e58ac24-9d1b-47a7-b0dd-98d7766bdc1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:10.979706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:11.072599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 109/219] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:11.073114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a11dc-7dda-4dcc-a4af-5743aa33853c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.073453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a11dc-7dda-4dcc-a4af-5743aa33853c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.073723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a11dc-7dda-4dcc-a4af-5743aa33853c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.075311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 110/220] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:11.081466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.081745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.082014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.107485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:11.107485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:11.108228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 111/221] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:11.112054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.215887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 111/222] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:11.303687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce1b0e3-9100-4496-82a3-34ee7b07346c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:11.324346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:11.358474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:11.359155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.359419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.359688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.360903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:11.362363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:11.364512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 112/223] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:11.371506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.400598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.424308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:11.559597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d182ab-2bd5-493e-b76c-93905d927885 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:11.582855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585139b3-e6ab-4f1e-92f5-8e4ed98595bf None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:11.601962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:11.601962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.601962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.601962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.602382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=11952b25a37d473b9c5591cced5dd912) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:11.602649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:11.604082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:11.618718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 112/224] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/role_assignments?user.id=11952b25a37d473b9c5591cced5dd912&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 259 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:11.619753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.644682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:11.646426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.647138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.647620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.664726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:11.671298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:11.672253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:11.673032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:11.675749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 113/225] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:11.688582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.778994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:11.832010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:11.839355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ce835f-0b91-4351-b71b-ff543311ac35 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:11.890061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:11.891136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.891672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.892144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.898489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88bb750-7069-40b1-9a0c-191ef4955067 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:11.908668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:11.914818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:11.915717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:11.916492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 113/226] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:11.921736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:11.975770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:11.976752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:11.977040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:11.977314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:11.978002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:11.978304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:11.979775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:11.992272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 114/227] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:11.997965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.121753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3b5da6-801b-4a52-840a-60e279bfc46f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.123665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:12.182738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.184190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.185618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.186296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.188361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e667bb-ff96-4baa-a7de-85c7dcc64a9b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.190227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.194404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.197386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 115/228] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:12.204653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.251351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.252141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.252499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.252869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.253604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:12.253960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.255621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.274691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 114/229] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:12.282021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.331002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:12.425738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c831f9d-53d1-44f4-b61e-d3e294f2d5d1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.457619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c62aa-df3a-4fb4-90c8-f507ea5bf8ed None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.498200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.498909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.499191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.499462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.500650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.502176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.504300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 116/230] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:12 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:12.510802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.535352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.536096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.536414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.536726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.543425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.545193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.547467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 115/231] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:12 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:12.559760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.564655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:12.673733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ad16f-71ab-4f8e-b343-67d2797d0a21 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.743939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:12.746090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.746816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.747097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.747370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/users/4a556d2b935246128748703dc8cf8002/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.751927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, user_id=4a556d2b935246128748703dc8cf8002, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.753433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.765972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461b58a7-d889-4241-afdb-a4bef7bd6259 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:12.777876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 117/232] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:21:12 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/users/4a556d2b935246128748703dc8cf8002/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:21:12.830283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:12.830967 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:12.831247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:12.831517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:12.832700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:12.834771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b988fa31-26d9-496f-8963-b7ebfe7ca97d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:12.837345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 116/233] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:12 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:12.843339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:12.987733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:12.992955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d5f4c-75e2-4e26-801e-78484dff64cc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:13.033165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:13.033748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:13.034024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:13.034297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/users/4a556d2b935246128748703dc8cf8002/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:13.036620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, user_id=4a556d2b935246128748703dc8cf8002, role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:13.038523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e9b3ee-dbdd-44d2-b1ba-61d6f7f9dfa9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:13.059116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 118/234] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:21:12 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/users/4a556d2b935246128748703dc8cf8002/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:13.604690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cd626-8392-47a5-91a7-2d8a24a93c49 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:13.605206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cd626-8392-47a5-91a7-2d8a24a93c49 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:13.605206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cd626-8392-47a5-91a7-2d8a24a93c49 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:13.606926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 117/235] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:13.610270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e35821-a32f-4cc8-8070-fe6968431c4f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:13.610978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e35821-a32f-4cc8-8070-fe6968431c4f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:13.613608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:13.614136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:13.614661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:13.619796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e35821-a32f-4cc8-8070-fe6968431c4f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:13.625220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 119/236] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:13 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:13.635702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:13.636591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:13.638213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:13.667758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:13.669322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:13.684453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:13.688310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:13.794339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 120/237] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:13.840929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 118/238] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:14.196761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:14.249606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:14.347077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:14.367495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5902567a-8f6f-44b6-be53-3dd504b42102 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:14.431985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:14.432606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.432876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.433140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.435457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:14.436964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:14.438745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 121/239] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:14.446504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:14.464529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:14.481843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a727952f-c2fb-4b25-82f8-59f23bb617fa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:14.533334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:14.534356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.534895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.535360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.569910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:14.577194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:14.578184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:14.579358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:14.589590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 119/240] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:14.599245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:14.661052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec0ce9-c914-429c-bae3-cdb0e15e040c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:14.711740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:14.712344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.712649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.712953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.714206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:14.716506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1155dbe9-b628-4a3c-9d8f-a5b5d429f407 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:14.726235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c9add9-b220-4167-9f3a-3c4a2c809af6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:14.749977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 122/241] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:14 2022] POST /identity/v3/users => generated 293 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:14.751263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b271b9-bd08-4ca9-a554-f34540bc1e4e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.752463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b271b9-bd08-4ca9-a554-f34540bc1e4e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.752746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b271b9-bd08-4ca9-a554-f34540bc1e4e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.757284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 123/242] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:14.762107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.762398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.762674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.800747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:14.801378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:14.801652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:14.801919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:14.802688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:14.803017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:14.805088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:14.807864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:14.809138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:14.821763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:14.826782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 120/243] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:14.835315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:14.855809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:14.943188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 124/244] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:15.029920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816bd514-902d-421c-82a5-12ddd7674287 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:15.075660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:15.076306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.076582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.076849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] PATH_INFO: `/v3/users/c8f57a34d8074d84a951161295dd6e65` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.102669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] RBAC: Authorizing `identity:get_user(user_id=c8f57a34d8074d84a951161295dd6e65)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:15.104169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:15.105879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 121/245] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:14 2022] GET /identity/v3/users/c8f57a34d8074d84a951161295dd6e65 => generated 297 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:15.112433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:15.237448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7718a7-c8b9-4c8b-b495-693c7e1b26e7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:15.283673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:15.284218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.284486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.284752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.298822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:15.333314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:15.371899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:15.373851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:15.431011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:15.484945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f51ff5f-eca3-4ceb-9af9-2f9a16353b83 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:15.522615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 125/246] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:15.529323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:15.529903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:15.530567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.530852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.531123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.546170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:15.555850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:15.556974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:15.558340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 122/247] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:15.566065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:15.599862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:15.699300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bcc808-9f39-44a4-ab8c-4385ada21627 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:15.724818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6395b-0ff7-4db3-9d23-defc42871a63 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:15.747557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:15.748599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.749070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.753199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.755533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:15.757762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a8024-5faa-4b96-bf9a-55aa05c4cfc8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:15.762305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 126/248] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:15 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:15.766241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48859ff-6824-485b-90f5-5c164e862e10 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.767638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48859ff-6824-485b-90f5-5c164e862e10 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.771207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48859ff-6824-485b-90f5-5c164e862e10 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.773009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 127/249] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:15.775585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48859ff-6824-485b-90f5-5c164e862e10 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:15.790246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:15.799858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:15.800399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.800722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.801002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.801678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:15.802036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:15.804923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c588266-3d04-4894-bbe0-8322c10ebdc5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:15.824519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 123/250] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:15.825808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:15.826095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:15.826367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:15.862184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:15.863793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:15.993118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48859ff-6824-485b-90f5-5c164e862e10 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:16.007532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 124/251] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:16.008445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:16.018985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:16.088849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:16.089434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.089707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.089981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.093192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=c8f57a34d8074d84a951161295dd6e65) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:16.093534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:16.096564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:16.109038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 128/252] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:15 2022] GET /identity/v3/role_assignments?user.id=c8f57a34d8074d84a951161295dd6e65&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 259 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:16.202857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1206bd-1324-45c0-8aa3-3cfca3fdf421 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:16.227681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:16.246504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:16.247240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.247516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.247786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] PATH_INFO: `/v3/users/11952b25a37d473b9c5591cced5dd912` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.249188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] RBAC: Authorizing `identity:get_user(user_id=11952b25a37d473b9c5591cced5dd912)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:16.250785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746af528-2b14-4bde-b4e5-bd0cbcdd47b1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:16.252958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 125/253] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:16 2022] GET /identity/v3/users/11952b25a37d473b9c5591cced5dd912 => generated 297 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:16.259716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:16.391889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6322cc5d-16b7-462d-9ce5-a73d478d69a4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:16.404580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc63786-3856-4a21-bd4b-e2f8820e9d09 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.404910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc63786-3856-4a21-bd4b-e2f8820e9d09 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.405115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc63786-3856-4a21-bd4b-e2f8820e9d09 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.406917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 126/254] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:16.413224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.413468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.413740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.447224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:16.448451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:16.474561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:16.476296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.476671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.477022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.478649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:16.480344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:16.482382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 129/255] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:16 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:16.488454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:16.567148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:16.662741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 127/256] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:16.663694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:16.694593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2a396-2107-40cf-a049-811aed5360b2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:16.776983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:16.777663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.777938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.778208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/users/11952b25a37d473b9c5591cced5dd912/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.784553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, user_id=11952b25a37d473b9c5591cced5dd912, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:16.786064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:16.814079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 130/257] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:21:16 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/users/11952b25a37d473b9c5591cced5dd912/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:21:16.826513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71288de7-650e-4384-8ffd-f39da1af9de6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:16.850251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:16.874773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:16.875553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:16.875877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:16.876255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:16.891471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:16.902644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:16.904804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:16.906951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 128/258] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:16.908536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.074254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eadc39-81a6-4d1a-b0ae-d854b8f054b4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.075873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:17.111740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.125595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:17.126143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:17.126446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:17.126779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:17.127543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:17.127842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:17.129307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:17.157198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 131/259] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:17.167078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.310976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e28ce-fa5c-405d-b752-f8bf9ac914f0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.337187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a559a-0131-4e87-ab0f-6954c31cc2d9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.352601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:17.353305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:17.353607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:17.353876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:17.355088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:17.357254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:17.359738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 132/260] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:17 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:17.369378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.411010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:17.411657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:17.411945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:17.412204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:17.439188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:17.445076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:17.445936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:17.446743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 129/261] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:17.453287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:17.454503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.537252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:17.624956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0c4c38-7f4b-4b81-abe6-905fe0740530 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.669630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb6640c-192c-4862-8a70-6c8a8906804c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.715297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:17.716291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:17.716792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:17.717263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:17.735392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:17.736129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:17.736475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:17.736743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:17.737606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:17.738273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:17.743833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:17.762778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 130/262] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:17.769363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:17.907703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:17.909563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:17.938532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b02a2-81b2-4d22-aba0-1232707821a5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:17.943216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:17.964435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:18.000230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:18.000887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.001162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.001432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] PATH_INFO: `/v3/users/4a556d2b935246128748703dc8cf8002` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.002625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] RBAC: Authorizing `identity:get_user(user_id=4a556d2b935246128748703dc8cf8002)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:18.004082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:18.006000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 131/263] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:17 2022] GET /identity/v3/users/4a556d2b935246128748703dc8cf8002 => generated 283 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:18.015949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:18.165744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 133/264] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 797 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:18.175645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae83ed-b89f-4a60-a828-6c1ea349f129 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:18.192112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:18.242329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:18.242972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.243454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.243891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.300151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:18.356675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:18.358150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0524b05c-4d59-4323-a53e-6efa5c6a36db None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:18.420978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f9bb95-e4e7-4d49-8bf1-9e12b71361f0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:18.472586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:18.482001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:18.483431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.483721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.484004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.488232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:18.489979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:18.491973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 134/265] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:18 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:18.499045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:18.573587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 132/266] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 558 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:18.574875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea89b-7d4f-414d-bf55-0f02e309a8c4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.575159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea89b-7d4f-414d-bf55-0f02e309a8c4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.575428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea89b-7d4f-414d-bf55-0f02e309a8c4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.579428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 133/267] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:18.581677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ea89b-7d4f-414d-bf55-0f02e309a8c4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:18.660625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:18.686486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f51831-9191-407c-8a5b-ef484b6071de None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:18.721530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ea89b-7d4f-414d-bf55-0f02e309a8c4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:18.766109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:18.766771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.767053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.767329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.768078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=4a556d2b935246128748703dc8cf8002) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:18.768387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:18.769878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15d506a-17fa-4be9-b0f7-fe1a29afbacb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:18.785604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 135/268] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:18 2022] GET /identity/v3/role_assignments?user.id=4a556d2b935246128748703dc8cf8002&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:18.786935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e72ff4-6957-42ba-976c-41271e71057a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.787224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e72ff4-6957-42ba-976c-41271e71057a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.787496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e72ff4-6957-42ba-976c-41271e71057a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.794235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:18.794925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:18.795231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:18.795481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:18.797013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:18.798531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:18.802740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 134/269] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:18 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:18.811946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:18.838056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c4e72ff4-6957-42ba-976c-41271e71057a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:18.841281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4e72ff4-6957-42ba-976c-41271e71057a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:18.974748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe10488-91e5-45c3-baa8-10ea9ba8f432 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:19.006066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:19.020303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:19.023330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.023766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.024208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.025097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=4a556d2b935246128748703dc8cf8002) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:19.025962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:19.028185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53215ae1-e1ce-4563-9c84-33ae51c1b1cd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:19.029911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 136/270] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:19.035835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb05b2-c80a-4c69-a924-20cb5fd828cb None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.036321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb05b2-c80a-4c69-a924-20cb5fd828cb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.036767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eb05b2-c80a-4c69-a924-20cb5fd828cb None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.039315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 137/271] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:19 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:19.043603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:19.045784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580bddf-c265-4a21-a59d-863ba4317f42 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.046234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580bddf-c265-4a21-a59d-863ba4317f42 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.046678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580bddf-c265-4a21-a59d-863ba4317f42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.051731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 135/272] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:18 2022] GET /identity/v3/role_assignments?user.id=4a556d2b935246128748703dc8cf8002&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 608 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:19.068729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f580bddf-c265-4a21-a59d-863ba4317f42 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:19.070228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f580bddf-c265-4a21-a59d-863ba4317f42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:19.185001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 138/273] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:19.290522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e8c661-c05c-4f43-8c88-c270ee7ae8e7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.290909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e8c661-c05c-4f43-8c88-c270ee7ae8e7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.291077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e8c661-c05c-4f43-8c88-c270ee7ae8e7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.292821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 136/274] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:19.298250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.298541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.298861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.323094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:19.324294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:19.405931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 139/275] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:19.474312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e8c661-c05c-4f43-8c88-c270ee7ae8e7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:19.559608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:19.610357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e8c661-c05c-4f43-8c88-c270ee7ae8e7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:19.653881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:19.654656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.654832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.655156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.656283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:19.657745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:19.663828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 137/276] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:19.667657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:19.669152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:19.730998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd9edf-3d91-48d5-9122-fdd0ff94e99d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:19.770180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:19.773738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:19.774540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.774915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.775398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.789534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:19.794632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:19.795741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 140/277] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:19.796676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:19.817333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd12a9-39c8-4f63-834c-29cf7743ca0d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:19.869414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:19.870035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.870310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.870601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.871249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:19.872694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:19.903125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 138/278] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:19 2022] POST /identity/v3/users => generated 293 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:21:19.904663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:19.931747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01054645-0253-42e6-995d-22ec656f7b2f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:19.949060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:19.976867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:19.977915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:19.978201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:19.978472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:19.991521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0d401fe3-13a8-45b4-b582-6f9350f5ccbd None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:19.996593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:19.997579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:19.998348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 141/279] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:19.999054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653ef6-c161-41b6-9d38-40716d753096 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.004471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653ef6-c161-41b6-9d38-40716d753096 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.004471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653ef6-c161-41b6-9d38-40716d753096 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.004471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 142/280] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:20.007130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63653ef6-c161-41b6-9d38-40716d753096 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.052530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99019ed-b44d-4d6d-b426-a3ff03849a05 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.086519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.102336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.102890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.103238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.103555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.104289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:20.104596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.106150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3663ce7-5beb-4ced-a1af-fbc2657f67a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.122541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 139/281] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.124169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.124883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.125407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.146729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63653ef6-c161-41b6-9d38-40716d753096 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.147784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:20.151659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:20.200366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.200993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.201318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.201550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.202875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:20.203180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.204584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.217751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 143/282] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.218742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.247097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 140/283] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:20.248538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.266973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.349929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d15fc-de2a-47fa-90f1-2fcce66fe811 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.389281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.389957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.390237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.390515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] PATH_INFO: `/v3/users/a76387cd16bf499aa7d55da20beea976` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.394283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab594957-d11a-4037-bf39-4044ae0546ae None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.408224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] RBAC: Authorizing `identity:get_user(user_id=a76387cd16bf499aa7d55da20beea976)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.408694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.410710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 144/284] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/users/a76387cd16bf499aa7d55da20beea976 => generated 293 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.419240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.434125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.473014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.473712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.473984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.474249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] PATH_INFO: `/v3/users/c8f57a34d8074d84a951161295dd6e65` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.477307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] RBAC: Authorizing `identity:get_user(user_id=c8f57a34d8074d84a951161295dd6e65)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.478926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.487624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 141/285] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/users/c8f57a34d8074d84a951161295dd6e65 => generated 297 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.498361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.587526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.594640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb743d-9522-4438-b14d-1d907e2b60bf None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.633578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.634187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.634469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.634785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.635519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, user.id=a76387cd16bf499aa7d55da20beea976, scope.system=all) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:20.635825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.637286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.643080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 145/286] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/role_assignments?user.id=a76387cd16bf499aa7d55da20beea976&scope.system=all&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.643912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.667739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225a6098-6a0f-455e-b6e4-0c3a5fda896d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.718111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.719130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.719407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.719676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.722012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:20.723585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:20.726392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 142/287] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:20.739956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.771304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbaa49f-94ed-43ee-8c27-3d31a120378a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:20.817166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:20.817821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:20.818103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:20.818427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:20.833480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:20.840955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:20.841904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:20.843629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 146/288] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:20.849685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:20.877119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.882910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:20.989834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be21a1c-0394-4ffd-b71c-16a5146e1a62 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:21.034378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b383f53-bf2d-4e9e-8e3b-64c554c5af8f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:21.079282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:21.080070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:21.080354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:21.080633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/users/c8f57a34d8074d84a951161295dd6e65/roles/b1f43cce7d8c45a6bea928386ae2bb04` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:21.087079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:21.087325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, user_id=c8f57a34d8074d84a951161295dd6e65, role_id=b1f43cce7d8c45a6bea928386ae2bb04)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:21.087973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:21.088376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:21.088767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:21.088880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:21.089756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:21.090209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:21.092938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:21.107047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 147/289] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:21.113580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:21.117551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 143/290] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:21:20 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/users/c8f57a34d8074d84a951161295dd6e65/roles/b1f43cce7d8c45a6bea928386ae2bb04 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:21.240497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:21.267542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c39278-3115-49c7-8f4b-344701f4566c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:21.314044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:21.314898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:21.315243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:21.315597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] PATH_INFO: `/v3/users/11952b25a37d473b9c5591cced5dd912` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:21.317181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] RBAC: Authorizing `identity:get_user(user_id=11952b25a37d473b9c5591cced5dd912)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:21.319149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:21.321756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 148/291] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:21 2022] GET /identity/v3/users/11952b25a37d473b9c5591cced5dd912 => generated 297 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:21.331825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:21.447426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459ed925-71e4-48a3-8bf1-8a3f58ee2a1f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:21.462912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:21.485376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:21.486005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:21.486329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:21.486629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:21.563400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:21.564797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:21.680734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 144/292] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:21.686437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:21.810198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460d6a15-67b1-4c76-9ac0-631bc718c616 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:21.857618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:21.858172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:21.858451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:21.862870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:21.864390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:21.865887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b47049-9678-4a85-bfe0-a7c5643e287b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:21.868026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 149/293] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:21 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:21.874289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:21.889693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:22.002286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113da46e-f140-42ef-83dc-5b456944442b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:22.047432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:22.048049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:22.048325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:22.048589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:22.049316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=11952b25a37d473b9c5591cced5dd912) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:22.049619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:22.051067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fd39ca-87fd-4e7e-94f3-bed5df5945be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:22.060221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 145/294] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:21 2022] GET /identity/v3/role_assignments?user.id=11952b25a37d473b9c5591cced5dd912&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:22.452885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e3559-47e8-4b63-af5e-9e607ab8bb2c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:22.453144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e3559-47e8-4b63-af5e-9e607ab8bb2c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:22.453510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e3559-47e8-4b63-af5e-9e607ab8bb2c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:22.454827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 150/295] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:22.461076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dda27-3cfc-4dbd-b79c-d42eb44bc21b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:22.461693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dda27-3cfc-4dbd-b79c-d42eb44bc21b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:22.461693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05dda27-3cfc-4dbd-b79c-d42eb44bc21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:22.485941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e05dda27-3cfc-4dbd-b79c-d42eb44bc21b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:22.487186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e05dda27-3cfc-4dbd-b79c-d42eb44bc21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:22.568492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 146/296] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:22.851440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e3559-47e8-4b63-af5e-9e607ab8bb2c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:22.872708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:22.967160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e3559-47e8-4b63-af5e-9e607ab8bb2c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.002123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.002786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.003064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.003333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.014661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:23.016082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1627d-1280-4913-ba80-963e43ade3c6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.016364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1627d-1280-4913-ba80-963e43ade3c6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.016635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b1627d-1280-4913-ba80-963e43ade3c6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.018237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 147/297] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:23.020117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:23.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:23.021762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 151/298] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:23.023631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.023910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.024179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.027719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.037291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.045134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:23.047426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:23.119050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.137415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e910b51-b233-4f25-855d-d7b58777ff79 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.146643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 148/299] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:23.190205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.191128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.191595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.192016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.192920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:23.193419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:23.195207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4788de5d-53bb-4f1a-b25e-ee933669a148 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:23.205780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 152/300] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:23.211442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.301927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd27999-a0e2-442a-ab13-45eddf8d6e55 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.302214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd27999-a0e2-442a-ab13-45eddf8d6e55 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.302494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd27999-a0e2-442a-ab13-45eddf8d6e55 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.304321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 153/301] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:23.309328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.309617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.309896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.318298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700a0081-b496-4b4e-9ab8-856895792aea None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.330918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:23.332414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:23.356660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.357606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.358026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.358580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] PATH_INFO: `/v3/users/72e50886aa4f44899d4fc505ac3a0698` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.362535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.377200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] RBAC: Authorizing `identity:get_user(user_id=72e50886aa4f44899d4fc505ac3a0698)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:23.379002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:23.381243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 149/302] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/users/72e50886aa4f44899d4fc505ac3a0698 => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:23.390599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.416730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 154/303] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:23.490670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca53e69-9f82-4f35-8303-d4eca0c8b93f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.492760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.527628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.528233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.528505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.528770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.529490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, user.id=72e50886aa4f44899d4fc505ac3a0698, scope.system=all) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:23.529792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:23.531265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:23.536645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 150/304] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/role_assignments?user.id=72e50886aa4f44899d4fc505ac3a0698&scope.system=all&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 226 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:23.572946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.616781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4bf791-033a-43cc-97a7-6a4ed5a31af6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.659569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.660132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.660405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.660673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.661831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:23.663702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:23.665791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 155/305] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:23.671711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.689210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:23.751034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.798647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e81be1-abfd-49a4-9a60-0c4d0f9fe08d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.834701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4972dbc7-bd68-4b72-9990-95f5ef78ccfe None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:23.845215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.846595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.847373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.847886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] PATH_INFO: `/v3/groups` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.848982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:23.852272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:23.874032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 151/306] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Oct 4 08:21:23 2022] POST /identity/v3/groups => generated 229 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:21:23.874816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:23.876101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:23.876555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:23.877033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:23.880984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:23.892438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:23.898745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:23.899728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:23.900543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:23.907713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 156/307] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:23.911321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:24.031710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f6f3e-275e-4b0a-ad03-c1ef3dfc97b4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:24.052134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:24.075732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:24.076617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.077043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.077464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.078308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:24.078954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:24.080544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e2d922-2567-44be-8d95-b40313a77185 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:24.096253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 152/308] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:24.097929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:24.211424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fdfc41-930b-41cd-a82c-ac05c4c7605f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.211718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fdfc41-930b-41cd-a82c-ac05c4c7605f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.212015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fdfc41-930b-41cd-a82c-ac05c4c7605f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.214049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 153/309] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:24.221605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.221954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.222281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.236633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7529a54b-f75a-4fd7-8221-6cecba887a7d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:24.250750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:24.252644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:24.276112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:24.277124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.277683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.278151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] PATH_INFO: `/v3/users/a76387cd16bf499aa7d55da20beea976` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.279612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] RBAC: Authorizing `identity:get_user(user_id=a76387cd16bf499aa7d55da20beea976)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:24.281366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:24.283620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 157/310] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:24 2022] GET /identity/v3/users/a76387cd16bf499aa7d55da20beea976 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:24.290375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:24.361345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:24.373354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 154/311] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:24.416312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd429e65-adcc-4353-9586-e6ebe1c7da9c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:24.452560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:24.454010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.454701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.455178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] PATH_INFO: `/v3/system/users/a76387cd16bf499aa7d55da20beea976/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.456841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=a76387cd16bf499aa7d55da20beea976, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:24.458461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:24.474822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 158/312] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Oct 4 08:21:24 2022] PUT /identity/v3/system/users/a76387cd16bf499aa7d55da20beea976/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:21:24.659527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:24.767790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe9a737-8f55-46da-ad9c-0f10008a781d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:24.779987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:24.805041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:24.806022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.806504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.807035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.818040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:24.823874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:24.824723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:24.825512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:24.828134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 155/313] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:24.836553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:24.951462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6600a65-b584-4521-a906-9ea3fe2442c6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:24.980393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:24.990764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:24.992531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:24.992814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:24.993087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:24.993753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:24.994053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:24.995472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:25.004190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 159/314] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:25.010730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:25.115449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b356e2-700b-4380-a588-ab9d1c127d50 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:25.157989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:25.158912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:25.159391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:25.159894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] PATH_INFO: `/v3/users/c8f57a34d8074d84a951161295dd6e65` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:25.161229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] RBAC: Authorizing `identity:get_user(user_id=c8f57a34d8074d84a951161295dd6e65)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:25.162940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:25.165708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 156/315] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:25 2022] GET /identity/v3/users/c8f57a34d8074d84a951161295dd6e65 => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:25.171829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:25.284540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf917b02-855a-410a-9783-f37616b6db07 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:25.320434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:25.321049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:25.321336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:25.321654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:25.390943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:25.399113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:25.400599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:25.510018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 160/316] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:25.515787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:25.561387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:25.626420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d7195-67b3-4932-879f-e1dd9714cd36 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:25.662484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:25.663226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:25.663564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:25.663987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:25.665148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:25.666585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a20382-c660-47ff-8f6f-963ec9957314 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:25.668565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 157/317] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:25 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:25.674102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:25.778090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:25.790395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0ebb7-cc86-4090-bd6e-6c23d31c5787 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:25.828291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:25.828978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:25.829296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:25.829605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:25.830503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, scope.project.id=c6aa628bef144413ae044679765bfb01, user.id=c8f57a34d8074d84a951161295dd6e65) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:25.830845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:25.832460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21265d59-3cf2-4a5b-bee2-d539c3a6bbb1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:25.840866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 161/318] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:25 2022] GET /identity/v3/role_assignments?user.id=c8f57a34d8074d84a951161295dd6e65&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:26.049034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bada7-e57e-446e-831f-7ea778c11b34 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.049685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bada7-e57e-446e-831f-7ea778c11b34 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.050260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bada7-e57e-446e-831f-7ea778c11b34 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.052333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 158/319] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:26.059229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d90a7c-bd50-4ef3-ad40-83f4f159587e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.059754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d90a7c-bd50-4ef3-ad40-83f4f159587e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.059754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d90a7c-bd50-4ef3-ad40-83f4f159587e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.082208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-50d90a7c-bd50-4ef3-ad40-83f4f159587e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:26.083557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-50d90a7c-bd50-4ef3-ad40-83f4f159587e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:26.173936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 162/320] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:26.196364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a8144-87f2-4903-b8d1-470c1833c73c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.196716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a8144-87f2-4903-b8d1-470c1833c73c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.197062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a8144-87f2-4903-b8d1-470c1833c73c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.202281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 159/321] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:26.203952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.204243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.204521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.222692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:26.223867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:26.298092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 163/322] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:26.478277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3a8144-87f2-4903-b8d1-470c1833c73c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:26.564773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:26.579396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:26.595890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3a8144-87f2-4903-b8d1-470c1833c73c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:26.605517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:26.635565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:26.636254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.636467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.636746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.648957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:26.654153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:26.655066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:26.655841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 160/323] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:26.661607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:26.681826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e72ef3-f401-4325-92af-961f08e4bbc5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:26.720056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:26.720710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.720987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.721293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.722451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:26.723985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:26.726617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 164/324] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:26.732027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:26.786757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266df97-145a-4473-9971-f12f14e3efb3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:26.821967 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:26.822584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.822864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.823151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.823634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:26.823843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:26.824325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:26.825484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:26.834192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 161/325] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:26.840757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:26.845022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb3e778-7481-43a3-b536-6fbd4f6d9c09 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:26.871066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:26.881913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:26.882632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.882922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.883195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] PATH_INFO: `/v3/groups` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.883826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:26.885269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9735596f-37a3-42eb-8d42-b8b0af9d7b14 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:26.902602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 165/326] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Oct 4 08:21:26 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:21:26.902804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01608ab9-2e19-4eb2-8f15-ac4908579f7c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.903033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01608ab9-2e19-4eb2-8f15-ac4908579f7c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.903368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01608ab9-2e19-4eb2-8f15-ac4908579f7c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.904513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 166/327] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:26.910339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:26.910647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:26.910924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:26.943373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:26.944930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:26.963705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a1541-027f-4a2d-850a-07b42e0cd599 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:27.005605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:27.006341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:27.006693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:27.007547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] PATH_INFO: `/v3/users/72e50886aa4f44899d4fc505ac3a0698` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:27.009235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] RBAC: Authorizing `identity:get_user(user_id=72e50886aa4f44899d4fc505ac3a0698)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:27.010924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:27.018110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 162/328] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:26 2022] GET /identity/v3/users/72e50886aa4f44899d4fc505ac3a0698 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:27.019226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:27.025329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 167/329] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:27.128089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:27.134422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b933a635-5d8a-4115-96a2-bbe1ff4f2b15 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:27.181171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:27.181803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:27.182079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:27.182349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] PATH_INFO: `/v3/system/users/72e50886aa4f44899d4fc505ac3a0698/roles/b1f43cce7d8c45a6bea928386ae2bb04` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:27.183812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=72e50886aa4f44899d4fc505ac3a0698, role_id=b1f43cce7d8c45a6bea928386ae2bb04)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:27.185246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:27.194779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 163/330] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Oct 4 08:21:27 2022] PUT /identity/v3/system/users/72e50886aa4f44899d4fc505ac3a0698/roles/b1f43cce7d8c45a6bea928386ae2bb04 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:27.304048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:27.389877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:27.427868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265caa4b-8952-450b-9e57-5dafd8db2516 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:27.469310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:27.470198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:27.470713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:27.471173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:27.484066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:27.489919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:27.491250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:27.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:27.495269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 168/331] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:27.500773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:27.640010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40295e78-dd64-45a7-bb16-38fe588a6a36 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:27.680833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:27.681447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:27.681727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:27.681996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:27.682723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:27.683022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:27.684393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:27.695400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 164/332] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:27.701401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:27.823140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1b704-091d-43e9-a3c5-f8c0e0ddbd44 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:27.856089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:27.863012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:27.863889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:27.864342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:27.865151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] PATH_INFO: `/v3/users/a76387cd16bf499aa7d55da20beea976` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:27.867043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] RBAC: Authorizing `identity:get_user(user_id=a76387cd16bf499aa7d55da20beea976)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:27.868830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c37d4d0-a210-498a-9868-72e49f11fa1a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:27.870651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 169/333] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:27 2022] GET /identity/v3/users/a76387cd16bf499aa7d55da20beea976 => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:27.877224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:27.904740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:27.996609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0588a4-0c51-4c3b-bc65-232a234d5030 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:28.041226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:28.042131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:28.042601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:28.043069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:28.043969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] RBAC: Adding query filter params (role.id=a0ac2c0678b9462799a770a2b667dd0e, user.id=a76387cd16bf499aa7d55da20beea976, scope.system=all) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:28.044450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:28.046072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cea20e2-564d-450f-a400-380abf73a16d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:28.065374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 165/334] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Oct 4 08:21:27 2022] GET /identity/v3/role_assignments?user.id=a76387cd16bf499aa7d55da20beea976&scope.system=all&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 510 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:29.680767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a845c37-997e-4d65-8ca5-e76db301aa7b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:29.681773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a845c37-997e-4d65-8ca5-e76db301aa7b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:29.682256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a845c37-997e-4d65-8ca5-e76db301aa7b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:29.684101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 170/335] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:29.690295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14099e0d-ba4c-438b-be7d-be175ec92db5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:29.691240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14099e0d-ba4c-438b-be7d-be175ec92db5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:29.691709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14099e0d-ba4c-438b-be7d-be175ec92db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:29.712809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-14099e0d-ba4c-438b-be7d-be175ec92db5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:29.714337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-14099e0d-ba4c-438b-be7d-be175ec92db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:29.805968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 166/336] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:30.064084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d990c-fb68-4379-90d5-aaf5d6dbffd8 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.064732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d990c-fb68-4379-90d5-aaf5d6dbffd8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.065212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73d990c-fb68-4379-90d5-aaf5d6dbffd8 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.067006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 171/337] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:30.074823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b114a-4764-4923-a057-fdd4f61347ac None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.075383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b114a-4764-4923-a057-fdd4f61347ac None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.075866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b114a-4764-4923-a057-fdd4f61347ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.103501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4c5b114a-4764-4923-a057-fdd4f61347ac None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:30.105041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4c5b114a-4764-4923-a057-fdd4f61347ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:30.109036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:30.197453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 167/338] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:30.366809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73d990c-fb68-4379-90d5-aaf5d6dbffd8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:30.434099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d270a-f106-417f-b3cf-11a9fec38b9a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.434372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d270a-f106-417f-b3cf-11a9fec38b9a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.434927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d270a-f106-417f-b3cf-11a9fec38b9a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.436750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 168/339] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:30.448943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758dc94-1095-40fd-abac-74360df5304b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.449548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758dc94-1095-40fd-abac-74360df5304b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.450017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7758dc94-1095-40fd-abac-74360df5304b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.474773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7758dc94-1095-40fd-abac-74360df5304b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:30.476915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7758dc94-1095-40fd-abac-74360df5304b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:30.557219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73d990c-fb68-4379-90d5-aaf5d6dbffd8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:30.600652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 169/340] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:30.606223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:30.607190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.607665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.608130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.621098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a6f9c17c-ab73-45d0-99ab-1d9d69a4f65d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:30.627049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:30.627992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:30.628777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:30.631312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 172/341] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:30.638839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758dc94-1095-40fd-abac-74360df5304b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:30.722963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:30.727116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a63cb-850d-461e-885c-aca46a9ae583 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.727937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a63cb-850d-461e-885c-aca46a9ae583 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.728416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280a63cb-850d-461e-885c-aca46a9ae583 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.729975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 173/342] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:30.735971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.736592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.737439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.768753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:30.769956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:30.775223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7758dc94-1095-40fd-abac-74360df5304b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:30.816555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:30.817529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:30.818046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:30.818514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:30.820477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:30.821055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:30.822693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:30.832673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 170/343] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:30.833728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:30.917605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:30.953197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 174/344] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:30.954321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:30.954979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af5e10-a2be-4c5e-94f5-bc3a43c352b1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:30.992817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.000699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.001647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.002135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.002674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.015525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:31.021480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:31.022399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:31.023813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:31.027085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 171/345] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:31.033022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.148273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b7872-e8c3-4108-9bed-10166cf9ffc4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.202663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968787ca-a8f5-417a-96c4-9e2facad9d3b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.219509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.220132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.220404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.221366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:31.221749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:31.223333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:31.241184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 172/346] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:31.242280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.276667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.277324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.277623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.277922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.348664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.384141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.403377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:31.405589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:31.414051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303c584-de3c-483a-b49f-a9266c0db971 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.461786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.462371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.462701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.462999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.476703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:31.485159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:31.487373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 173/347] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:31.489398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.546312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 175/348] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 592 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:31.547127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.631612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a66677-c76d-4f63-b358-fbfb93943735 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.650815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.663407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.675302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75567b80-2923-4ae7-a821-6350a1b844f9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.683281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.683901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.684698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.684972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.715956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.716594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.716900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.717236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.731765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:31.737537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:31.740274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 176/349] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:31.742689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.771782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:31.773922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:31.869688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7ee576-abff-4221-bcd8-58bd1420da9c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:31.902108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 174/350] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:31.903402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.910147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:31.910875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:31.911181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:31.911482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:31.912216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:31.912540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:31.914691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:31.921296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:31.924464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 177/351] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:31.931052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:31.949717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.052373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad920b1-0f9e-4215-911a-afd5f1ef1128 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.059471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5d9df4-9edf-4da2-80e0-94e4416d5ca5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.104961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.105865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.106144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.106412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.106887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.107126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:32.107430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.107599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.107861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.108135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.108818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.109323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.110980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.113481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 175/352] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.114086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.124346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 178/353] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.125076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.222022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.229129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.256042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebeceb5-c1ce-447f-ab20-0e475453b40d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.256230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea408d1e-d83e-4530-983d-998062ec0e7f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.301435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.302066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.302345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.302697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] PATH_INFO: `/v3/users/72e50886aa4f44899d4fc505ac3a0698` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.303785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] RBAC: Authorizing `identity:get_user(user_id=72e50886aa4f44899d4fc505ac3a0698)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.306152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a85bbf-720b-446b-b29c-3208e3d6efdb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.307878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 179/354] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/users/72e50886aa4f44899d4fc505ac3a0698 => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.309089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44db296-b29d-4df2-98d1-d22f83cdad33 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.309371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44db296-b29d-4df2-98d1-d22f83cdad33 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.309652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44db296-b29d-4df2-98d1-d22f83cdad33 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.310865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 180/355] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:32.311552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44db296-b29d-4df2-98d1-d22f83cdad33 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.312866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.314160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.314508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.314821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.316202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.317624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.319433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 176/356] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.320192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.435602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44db296-b29d-4df2-98d1-d22f83cdad33 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.447629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01537058-dcd0-43a7-b10d-d626a1cd36be None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.477809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.478814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.479399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.479674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.487735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.489263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143d3e71-4ac4-4659-9832-077ee3f8ea63 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.490414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.491079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.491195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 181/357] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.491494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.491788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.492616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22403982-95d1-4472-8318-35df48e4663d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.493004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22403982-95d1-4472-8318-35df48e4663d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.493281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22403982-95d1-4472-8318-35df48e4663d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.495319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 182/358] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:32.496062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22403982-95d1-4472-8318-35df48e4663d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.506645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.519722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.580855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.582364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.616520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22403982-95d1-4472-8318-35df48e4663d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.655073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.655689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.655976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.656235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.656972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] RBAC: Adding query filter params (role.id=b1f43cce7d8c45a6bea928386ae2bb04, user.id=72e50886aa4f44899d4fc505ac3a0698, scope.system=all) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:32.657292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.659254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca858c68-0146-4965-b266-79ce63d3197e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.676330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 183/359] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/role_assignments?user.id=72e50886aa4f44899d4fc505ac3a0698&scope.system=all&role.id=b1f43cce7d8c45a6bea928386ae2bb04 => generated 510 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.677620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.677868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.678520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.702276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:32.703585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:32.715014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 177/360] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:32.716116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.791200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.792065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:32.810371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 184/361] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:32.811145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:32.866772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef60da-d9bd-4277-84fe-d67db7c395dd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.926503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:32.927533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.927874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.928149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.930380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:32.931877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e74b757-c32c-4786-bffb-46851127ee4a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:32.934802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 178/362] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:32.936011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:32.936286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:32.936554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:32.956491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123dadd-e1fd-4598-ab6c-5fbeca8b74d9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:32.957570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:32.958780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:33.002076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.002897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.003003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.003370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.004255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:33.004255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.005687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.012437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 185/363] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:32 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 260 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.013133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.058752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 179/364] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:33.059858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.086039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.134093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65751bd3-a30a-4299-8936-a10fe7d785eb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.162513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.181415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.182804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.183233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.183899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.185287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.187234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.190022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 186/365] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.191056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.234168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ed9fc-08f2-4dea-b87b-60dbd9694d99 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.286587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.287605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.288070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.288945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.289688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:33.289994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.293114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.300450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 180/366] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 260 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.300450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.335098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb9d2f0-6253-47fd-8259-5c24d8e591fc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.390012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.411093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.411742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.412017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.412284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.428199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:33.435316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:33.436275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:33.437076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 187/367] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:33.438318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.474305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537a97e3-e31f-49f5-bb67-cdf53a995e18 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.496880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.524512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.525213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.525492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.525757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.526921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.528352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.530052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 181/368] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.530899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.562192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131dd6e8-c1df-4478-842b-dea4c512eb29 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.602472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.603144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.603568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.603849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.619204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:33.625488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:33.626354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:33.627150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 188/369] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:33.628363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.661515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd135d2-d24d-488d-a1f0-4f1459ea322f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.690781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.713462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.714179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.714527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.714897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.715717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:33.716089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.717581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.732956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 182/370] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.733963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.768527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72957320-4a29-4f88-a107-64fdc3d6ef19 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.797676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.812601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.813234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.813489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.813745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.814485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=c6aa628bef144413ae044679765bfb01) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:33.814819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.816279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.824820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 189/371] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 260 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.825851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.869700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e05d-1a56-4913-bf76-ac8abe643de1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.910517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.911160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.911439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.911764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:33.912378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:33.912681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:33.915056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:33.928117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 183/372] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:33.935579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:33.951054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40871ed8-e7bb-4f41-8e22-343360b2554f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:33.963434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:33.992155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:33.992829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:33.993176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:33.993456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.080546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152632e9-3ea8-4c35-8ada-fa314637eec8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.087722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:34.097378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.098909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.162148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.164672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.164672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.164672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.279482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 190/373] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:34.283408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.284994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.287677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:34.307117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:34.473025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763de106-a011-4fb5-b41c-5f7bf887c5cd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.499573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 184/374] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:21:34.505096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:34.509986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:34.541621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.542310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.542607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.542881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.544180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.545638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.549645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 191/375] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:34 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:34.554785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:34.654836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:34.690310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec90c6b-247a-4d17-8a1d-d273c5bdbcfe None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.731991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67b9e6f-a752-4359-b4a6-b6a3acc73cb4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.753897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.754715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.755091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.755460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.758935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.760510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.764520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 192/376] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:34 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:34.768062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:34.778338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.779249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.779681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.780092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] PATH_INFO: `/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.781385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] RBAC: Authorizing `identity:get_project(project_id=dc7bdc31e1404eecb2eaeb287aaeab06)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.783017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.785134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 185/377] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:34 2022] GET /identity/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06 => generated 344 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:34.791306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:34.899462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:34.903708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893be65f-af50-4797-a7a4-51d9b4e748f9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.942835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c253b2c-3c6c-4a8d-9f43-938fa3d470a7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:34.943426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.944328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.945549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.946095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:34.947442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=c6aa628bef144413ae044679765bfb01) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:34.947913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:34.949635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:34.955905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 193/378] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:34 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:34.993776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:34.994464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:34.994823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:34.995145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] PATH_INFO: `/v3/groups/580e4775e71543b7be2d5b4ab15910f4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:35.003461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] RBAC: Authorizing `identity:get_group(group_id=580e4775e71543b7be2d5b4ab15910f4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:35.005234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cc5eed-3b43-47fa-8004-b831643c7fa1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:35.008059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 186/379] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:34 2022] GET /identity/v3/groups/580e4775e71543b7be2d5b4ab15910f4 => generated 229 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:35.015259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:35.047616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:35.160645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7b831-0cb0-43a8-8a75-bb83d82b7e00 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:35.219395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:35.220066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:35.220387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:35.220750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:35.221643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] RBAC: Adding query filter params (group.id=580e4775e71543b7be2d5b4ab15910f4, role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=dc7bdc31e1404eecb2eaeb287aaeab06) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:35.222009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:35.223674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48136d0-f4e0-476e-8f68-9abd654d1b4a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:35.230811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 194/380] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:35 2022] GET /identity/v3/role_assignments?group.id=580e4775e71543b7be2d5b4ab15910f4&scope.project.id=dc7bdc31e1404eecb2eaeb287aaeab06&role.id=d7f1381a769e433aa1b033022931be96 => generated 260 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:35.870486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb10f1-0eb5-4fdc-9c8d-e3b71556a74f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:35.872427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb10f1-0eb5-4fdc-9c8d-e3b71556a74f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:35.872427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bb10f1-0eb5-4fdc-9c8d-e3b71556a74f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:35.872964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 187/381] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:35.878508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374a308f-20f7-48dc-b129-5eee8c0109ba None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:35.879046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374a308f-20f7-48dc-b129-5eee8c0109ba None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:35.879528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374a308f-20f7-48dc-b129-5eee8c0109ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:35.901939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-374a308f-20f7-48dc-b129-5eee8c0109ba None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:35.903561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-374a308f-20f7-48dc-b129-5eee8c0109ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:35.987063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 195/382] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:36.049289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba87956-6a23-4f0c-bff5-a2c7d750fc58 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.050225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba87956-6a23-4f0c-bff5-a2c7d750fc58 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.050714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba87956-6a23-4f0c-bff5-a2c7d750fc58 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.052570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 188/383] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:36.060011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.060522 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.060990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.068763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:36.081633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:36.082839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:36.174331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 196/384] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:36.386153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba87956-6a23-4f0c-bff5-a2c7d750fc58 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:36.517135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba87956-6a23-4f0c-bff5-a2c7d750fc58 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:36.552132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:36.553099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.553616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.554900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.557900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:36.566742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:36.572795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:36.573762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:36.574628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 189/385] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:36.579631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:36.687002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:36.767610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df442899-a179-4bf6-be82-9fb6257013e7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:36.843664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:36.844791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.845309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.845836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.846905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:36.847478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:36.849240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:36.851791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:36.862861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 197/386] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:36.869315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:36.874901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f05314-c3dc-4e58-98aa-36d1e46b86a5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:36.934080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:36.935273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.935807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.936331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.952073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-26ec415e-7115-4a21-b45e-4d6f27126535 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:36.959646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:36.960773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:36.962037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:36.964738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 190/387] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:36.970581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d09d27-4402-4679-82fe-097dbd458f3f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:36.974028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d09d27-4402-4679-82fe-097dbd458f3f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:36.974598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d09d27-4402-4679-82fe-097dbd458f3f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:36.976829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 191/388] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:36 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:36.977996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d09d27-4402-4679-82fe-097dbd458f3f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.026783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd22e2-56f8-47f3-8370-624b096f4d72 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.056103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.067376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.068283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.068703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.069117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.070676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.072272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b9cfb7-ddab-424a-9fbd-b720fd2827da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.078648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 198/389] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:36 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:37.080195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.080620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.081033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.102193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:37.103258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d09d27-4402-4679-82fe-097dbd458f3f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.103988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:37.138692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.139306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.139579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.145286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.146199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:37.146714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.149061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.164777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 192/390] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:37.166003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.168806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.179912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 199/391] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:37.186711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.271771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b868be4-c8af-43f2-8e04-5bfa5228c0f0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.291002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.316538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.317533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.318012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.318480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.320153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.323105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.325686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 193/392] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:37 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:37.332012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.420484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5376f25-25e7-43d5-9bea-4d88f7bd4dd8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.487333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.487982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.488258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.488530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.495389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.496860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.497059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828dbdea-bd73-426d-ab37-806b1404b979 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.498684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 200/393] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:37 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:37.504257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.534494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.535407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.535916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.536424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/groups/c46eb24e28274f7caeb16f07e70eac15/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.539540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, group_id=c46eb24e28274f7caeb16f07e70eac15, role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.541388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.559741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 194/394] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Oct 4 08:21:37 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/groups/c46eb24e28274f7caeb16f07e70eac15/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:21:37.561040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.581092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.619067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.635990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c210055b-c64d-4b2b-88ec-84c6516a3a24 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.678706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.680376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.680797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.681206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.682811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.684460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.685765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb75d23-2b8b-48e5-9833-6dfe8dc4e8bf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.686991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 201/395] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:37 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:37.697264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.738488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.739199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.739478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.739747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.761345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:37.766320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:37.767272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:37.768130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 195/396] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:37.777986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:37.861349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae14f9d-ea05-4273-8fd0-661309680d62 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.907274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.908346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.908821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.909290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/groups/c46eb24e28274f7caeb16f07e70eac15/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.910876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.912721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] RBAC: Authorizing `identity:create_grant(project_id=7fea84d924b34feb92e0e8acefe7e93d, group_id=c46eb24e28274f7caeb16f07e70eac15, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.914915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db1b05-13d9-4d65-8876-48520669203a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.929676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa35b17c-1212-4318-b2d9-25507f65d8d7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:37.933345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 202/397] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Oct 4 08:21:37 2022] PUT /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d/groups/c46eb24e28274f7caeb16f07e70eac15/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:37.939106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:37.959762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4644-3508-47bc-9489-7c5cddd87c15 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.961488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4644-3508-47bc-9489-7c5cddd87c15 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.961989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9c4644-3508-47bc-9489-7c5cddd87c15 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.964796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 203/398] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:37.971107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.971107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.971107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.983481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:37.984411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:37.985440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:37.985872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:37.987040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:37.987505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:37.989021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:37.991165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:37.992470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:38.001397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 196/399] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:38.007823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:38.124207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 204/400] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:38.171778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a340e0-4612-4a30-be89-16fb0a002b6f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:38.221158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:38.221802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.222080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.222353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.223523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:38.225062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:38.227573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 197/401] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:38 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:38.233221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:38.344140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc73d90-32c1-4518-b2a1-39aca11cfae5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:38.367581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:38.382988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:38.383699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.383979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.384253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.385396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:38.386874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff63e83b-5202-4408-98f4-d6b915180c3c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:38.388820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 205/402] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:38 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:38.395614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:38.446105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:38.457901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e00e6b-a44d-42f0-ba67-e045a5c48db0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.458189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e00e6b-a44d-42f0-ba67-e045a5c48db0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.458450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e00e6b-a44d-42f0-ba67-e045a5c48db0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.459967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 206/403] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:21:38.472196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.472473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.472758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.493833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:38.495541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:38.528265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef47b8d-4bbe-49dc-a554-8f4e68a26954 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:38.568428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:38.569086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.569360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.569627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/groups/c46eb24e28274f7caeb16f07e70eac15/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.572433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, group_id=c46eb24e28274f7caeb16f07e70eac15, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:38.573916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:38.588886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 198/404] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Oct 4 08:21:38 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/groups/c46eb24e28274f7caeb16f07e70eac15/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:38.589869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:38.591895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 207/405] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:38.699072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:38.703286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8453bf87-1001-4f16-85a5-bc338ab666d6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:38.741873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:38.742727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:38.743043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:38.743401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:38.755326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:38.761448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:38.762386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:38.763202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:38.766706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 199/406] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:38.776130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:38.940085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48977f06-7dbd-46ae-83cf-06f9232763c0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:38.971714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:38.989793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.006381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.007022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.007849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.008119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.008839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:39.009134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.011814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.029856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 208/407] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:39.036614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.197573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18412b3-6221-4ed6-a099-10432855b43a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.222093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2acfa6-10fe-4780-b6c6-b2d4257aa6b8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.255794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.256812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.257316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.257819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.268922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:39.275242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:39.276106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:39.276890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:39.279640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 200/408] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:39.287710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.300097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.300748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.301079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.301355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.302586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.304145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.306902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 209/409] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:39 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:39.311636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.319968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:39.368820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:39.440616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d183c3-e5d6-477e-891e-5eab8759a1f4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.445426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc61085-cbc3-410c-a3f3-96152c05bb59 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.478296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.478901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.479182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.479497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.480655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.482181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.484244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 210/410] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:39 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:39.489279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.489899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.490182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.490750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.491466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:39.491859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.493392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.495604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.511017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 201/411] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:39.524314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.609986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9bed0e-71cd-441e-ba1a-3583177fe2ae None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.650929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:39.651117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:39.672465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0458f8-0835-4730-830d-8e51070785dc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.673554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.674221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.674501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.674862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01/groups/c46eb24e28274f7caeb16f07e70eac15/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.677266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] RBAC: Authorizing `identity:create_grant(project_id=c6aa628bef144413ae044679765bfb01, group_id=c46eb24e28274f7caeb16f07e70eac15, role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.678789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.694955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 211/412] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Oct 4 08:21:39 2022] PUT /identity/v3/projects/c6aa628bef144413ae044679765bfb01/groups/c46eb24e28274f7caeb16f07e70eac15/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:39.766305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.766897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.767170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.767434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] PATH_INFO: `/v3/groups/580e4775e71543b7be2d5b4ab15910f4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.771373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] RBAC: Authorizing `identity:get_group(group_id=580e4775e71543b7be2d5b4ab15910f4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.773018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.775098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 202/413] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:39 2022] GET /identity/v3/groups/580e4775e71543b7be2d5b4ab15910f4 => generated 229 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:39.781211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:39.939998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b8b502-57ae-417d-99c0-f8b1d7136375 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:39.977986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:39.978715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:39.979037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:39.979337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] PATH_INFO: `/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:39.980502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] RBAC: Authorizing `identity:get_project(project_id=dc7bdc31e1404eecb2eaeb287aaeab06)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:39.982039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebdc462-aaa0-4d4b-9d15-5424e5167618 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:39.984099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 212/414] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:39 2022] GET /identity/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06 => generated 344 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:39.990112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:40.105351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04b6085-2007-44c1-bd0b-f9645551bed6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:40.143965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:40.143965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:40.143965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:40.143965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] PATH_INFO: `/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06/groups/580e4775e71543b7be2d5b4ab15910f4/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:40.145692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] RBAC: Authorizing `identity:create_grant(project_id=dc7bdc31e1404eecb2eaeb287aaeab06, group_id=580e4775e71543b7be2d5b4ab15910f4, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:40.149087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e85ac81-1d07-476c-97aa-cc2cdf5be269 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:40.167412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 203/415] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Oct 4 08:21:39 2022] PUT /identity/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06/groups/580e4775e71543b7be2d5b4ab15910f4/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:21:40.439252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c2f44-55da-4178-8204-c822a0e9ae8d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:40.439779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c2f44-55da-4178-8204-c822a0e9ae8d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:40.439779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98c2f44-55da-4178-8204-c822a0e9ae8d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:40.441710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 213/416] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:40.448099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b013efc-93f6-40f9-b084-d7d5c49c92a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:40.448567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b013efc-93f6-40f9-b084-d7d5c49c92a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:40.448667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b013efc-93f6-40f9-b084-d7d5c49c92a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:40.463689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:40.470629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7b013efc-93f6-40f9-b084-d7d5c49c92a0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:40.472568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7b013efc-93f6-40f9-b084-d7d5c49c92a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:40.559459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 204/417] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:40.601725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b305a-e7ff-466c-ae55-66a930ce84b9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:40.602017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b305a-e7ff-466c-ae55-66a930ce84b9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:40.602275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b305a-e7ff-466c-ae55-66a930ce84b9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:40.604104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 214/418] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:40.609768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:40.610103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:40.610403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:40.630739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:40.631987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:40.707297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 205/419] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:41.000125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b305a-e7ff-466c-ae55-66a930ce84b9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.017385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.071740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.099890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.144656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a64d5-dc23-4ca3-9908-7a017c28deb4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.185247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.186467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.186915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.187354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.199887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:41.205468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:41.206657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:41.207720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:41.210223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 206/420] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:41.220894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.221282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b305a-e7ff-466c-ae55-66a930ce84b9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.278977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.279945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.280415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.280881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.292562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:41.298288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:41.299217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:41.300005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:41.302621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 215/421] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:41.309745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.434228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fc0b48-f8f6-43cc-9c8e-82e5ef6a6199 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.493401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.501272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09739ad5-0368-4031-9c1c-4aa3e462131b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.511604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.512222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.512557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.512831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.513503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:41.513807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:41.515280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:41.519075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.534578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 207/422] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:41.535451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.574111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.574753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.575037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.575309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.576006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:41.576352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:41.577797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:41.598650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 216/423] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:41.605261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.673986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12493b0-a9f8-4a1c-953d-b4c0315f6150 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.712521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.713151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.713431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.713701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.782957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.791442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:41.792980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30787e95-12ba-4dc6-a1c5-015d54bed68a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:41.815208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7db0638-5377-4ebb-bad9-e01ef1384864 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:41.863748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:41.865271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.865986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.866658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.905170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:41.921406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 208/424] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:41.927862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13e11e9-e9d6-4f5f-abbc-347199f779e6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:41.928286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13e11e9-e9d6-4f5f-abbc-347199f779e6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:41.928722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13e11e9-e9d6-4f5f-abbc-347199f779e6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:41.931218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 209/425] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:41.932101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13e11e9-e9d6-4f5f-abbc-347199f779e6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:41.953364 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:41.955153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c08c4a2-07f8-491e-b4db-0b71478de389 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.031759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.046080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13e11e9-e9d6-4f5f-abbc-347199f779e6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.074972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 217/426] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:42.076294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.076575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.076842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.083636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.084566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.085019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.085558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.086982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.088733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.090577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 210/427] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:41 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:42.091527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.104317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:42.105602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:42.181644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.191452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 218/428] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:42.192355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.210056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea303b-8dcb-4e9b-ae92-5c3eaaaf6597 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.247766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.248767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.249243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.249708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] PATH_INFO: `/v3/projects/7fea84d924b34feb92e0e8acefe7e93d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.251151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] RBAC: Authorizing `identity:get_project(project_id=7fea84d924b34feb92e0e8acefe7e93d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.252882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.254819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 211/429] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/projects/7fea84d924b34feb92e0e8acefe7e93d => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:42.268061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.311959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a55ec2-9434-4850-bf8a-bbdab98b306e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.312277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.353863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.354560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.354843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.356511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.358385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.359855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.361592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 219/430] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:42.375165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.446112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd50270-aca0-4cc9-b945-d49d96daf910 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.498914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.500550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.501313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.502827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.503308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.504713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.507099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e50432a-4dca-4813-b37a-a70421a99250 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.510256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 212/431] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:42.512006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ec00d-45f1-47f8-80d9-b33828cf58cb None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.512758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ec00d-45f1-47f8-80d9-b33828cf58cb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.514406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76ec00d-45f1-47f8-80d9-b33828cf58cb None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.516129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 213/432] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:42.517055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76ec00d-45f1-47f8-80d9-b33828cf58cb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.565339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff56217e-501f-4fbb-b4c1-5775a407e935 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.607451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.608057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.608327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.608592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.609308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:42.609605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.611074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacb17de-51bd-46c4-9840-f0fcfb4b4738 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.617449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 220/433] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 611 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:42.618773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7ba2d-ea7a-4628-b06d-9f44285387d2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.619049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7ba2d-ea7a-4628-b06d-9f44285387d2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.619314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7ba2d-ea7a-4628-b06d-9f44285387d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.634861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.640433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-23e7ba2d-ea7a-4628-b06d-9f44285387d2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:42.641651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-23e7ba2d-ea7a-4628-b06d-9f44285387d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:42.645998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76ec00d-45f1-47f8-80d9-b33828cf58cb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.693743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.694109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.694323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.694663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.695620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=7fea84d924b34feb92e0e8acefe7e93d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:42.695924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:42.697499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:42.714972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 214/434] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=7fea84d924b34feb92e0e8acefe7e93d&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 611 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:42.714972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:42.738655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 221/435] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:42.740170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc328d-09dd-4948-92c7-4165bd75ea2a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.740516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc328d-09dd-4948-92c7-4165bd75ea2a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.741107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dc328d-09dd-4948-92c7-4165bd75ea2a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.743970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 222/436] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:42.764682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.764988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.765307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.808504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:42.809801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:42.893078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:42.918647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 223/437] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:42.926149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57d8b39-dd8f-401e-b060-2c1d0820b00e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:42.965001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:42.965902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:42.966342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:42.966653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:42.977181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:42.982748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:42.983633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 04 08:21:42.984404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:42.985507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 215/438] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:42.991572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.090246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.101019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd0912f-3251-4d81-8b8a-69740602032d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.112508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.151578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.152512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.155624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.156145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.157140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:43.157689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.159490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.170428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 224/439] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:21:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:43.176417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.238330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74becdf0-6f88-45f1-9879-d80fe9863b07 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.244934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.278477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.279157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.279439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.279700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.292208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:43.297451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:43.298499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 216/440] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:43.299427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.314456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a392f2c8-a7d1-4040-aba5-091f0eeed6a2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.357131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.357131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.357131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.357131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.394699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.421585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7978dc5-2116-4a9c-b16a-3b2c5a81481f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.439072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.440651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.455507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.456182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.456465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.456730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.467647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:43.472419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:43.473444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 217/441] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:43.474406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.495083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.550981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 225/442] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 04 08:21:43.550981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.579525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3111da-d604-48b2-8d38-7dfe204307ac None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.614375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.614993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.615265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.615527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.616430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:43.616723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.618096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.620235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.626940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 218/443] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:43.627733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.655575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e732e-056b-4b6f-84e1-0ff830719631 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.692928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.693614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.693889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.694255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.695102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:43.696014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.697560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.706937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 226/444] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:43.707920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.724670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.737177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e6f0c1-9d47-430a-aabe-51dd802a84aa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.770407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.771053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.771324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.771589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.772746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.775345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.776946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 219/445] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:43.777689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:43.818802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a1aac-cfbb-4f07-b12d-e47eacdaba27 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.841464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.855151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.855755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.856043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.856341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.877194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5deb2b-e7a0-4632-856d-d00323af0cef None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:43.911803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:43.912398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:43.912692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:43.912998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:43.930821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.932281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:43.946885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:43.989356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:43.991090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.044968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 227/446] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:44.044968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.065398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.102388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 220/447] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:44.103243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.146484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52105e21-2c35-471d-9163-0d3929dbf1ba None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.161546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.184369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.184990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.185262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.185532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.186652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.188092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.189632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 228/448] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.190462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.210449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0394873-3097-45d1-9125-237c18aa7e55 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.244375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.245005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.245280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.245551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] PATH_INFO: `/v3/projects/c6aa628bef144413ae044679765bfb01` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.246694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] RBAC: Authorizing `identity:get_project(project_id=c6aa628bef144413ae044679765bfb01)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.248137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.249750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 221/449] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/projects/c6aa628bef144413ae044679765bfb01 => generated 302 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.250527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.284570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.293727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce58f41d-1a0c-4bd3-bfc7-2d915b4131e1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.326969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.327590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.327866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.328135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] PATH_INFO: `/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.329241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] RBAC: Authorizing `identity:get_project(project_id=dc7bdc31e1404eecb2eaeb287aaeab06)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.330725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.332339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 229/450] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/projects/dc7bdc31e1404eecb2eaeb287aaeab06 => generated 344 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.338296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.357989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22bc0f5-9c0c-4a39-b942-8f8fa383c918 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.387966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.392348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.392944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.393229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.393475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.394190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=a0ac2c0678b9462799a770a2b667dd0e, scope.project.id=c6aa628bef144413ae044679765bfb01) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:44.394486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.395933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.402055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 222/451] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=a0ac2c0678b9462799a770a2b667dd0e => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.414982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.456441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d818441-2e65-451a-98b0-6534e5a8bb20 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.500369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.500939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.501388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.501665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] PATH_INFO: `/v3/groups/c46eb24e28274f7caeb16f07e70eac15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.503593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] RBAC: Authorizing `identity:get_group(group_id=c46eb24e28274f7caeb16f07e70eac15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.505247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.507450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 230/452] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/groups/c46eb24e28274f7caeb16f07e70eac15 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.513781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.522656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.529744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a228a-dd10-4bc0-8d1e-e572d6659b92 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.573007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.573007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.573308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.574067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] PATH_INFO: `/v3/groups/580e4775e71543b7be2d5b4ab15910f4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.575260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] RBAC: Authorizing `identity:get_group(group_id=580e4775e71543b7be2d5b4ab15910f4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.577474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.582632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 223/453] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/groups/580e4775e71543b7be2d5b4ab15910f4 => generated 229 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:44.584929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:44.640523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36d5b2-0f72-4050-9bf0-76167edb901a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.660524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:44.684528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.685137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.685407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.685672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.686697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] RBAC: Adding query filter params (group.id=c46eb24e28274f7caeb16f07e70eac15, role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=c6aa628bef144413ae044679765bfb01) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:44.686801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.688222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6690b5-6eda-49c9-8e2a-4a8cd791ffa4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.694924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 231/454] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/role_assignments?group.id=c46eb24e28274f7caeb16f07e70eac15&scope.project.id=c6aa628bef144413ae044679765bfb01&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 611 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:44.717978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad10e71-4bb0-4e26-b86d-b505efbc7b8c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:44.767580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:44.768117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:44.768392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:44.768658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:44.769574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] RBAC: Adding query filter params (group.id=580e4775e71543b7be2d5b4ab15910f4, role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=dc7bdc31e1404eecb2eaeb287aaeab06) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:44.769687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:44.771187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be78d9a2-5289-468e-a1f9-6c50bdcdcfdc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:44.778814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 224/455] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Oct 4 08:21:44 2022] GET /identity/v3/role_assignments?group.id=580e4775e71543b7be2d5b4ab15910f4&scope.project.id=dc7bdc31e1404eecb2eaeb287aaeab06&role.id=d7f1381a769e433aa1b033022931be96 => generated 611 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:47.614449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b0565-a01e-41d0-9e6e-17075d922246 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:47.614968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b0565-a01e-41d0-9e6e-17075d922246 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:47.614968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b0565-a01e-41d0-9e6e-17075d922246 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:47.616820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 232/456] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:47.623619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb450-97cf-415f-a555-74e65699cf9b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:47.624052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb450-97cf-415f-a555-74e65699cf9b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:47.624227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5fb450-97cf-415f-a555-74e65699cf9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:47.651738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ea5fb450-97cf-415f-a555-74e65699cf9b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:47.653086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ea5fb450-97cf-415f-a555-74e65699cf9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:47.688082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:47.745446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 225/457] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:47.792863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab9bd40-7ce6-4275-b128-5a0ab3aa2fa6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:47.793169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab9bd40-7ce6-4275-b128-5a0ab3aa2fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:47.793416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab9bd40-7ce6-4275-b128-5a0ab3aa2fa6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:47.795136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 233/458] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:47.800029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e45962-3574-46d8-b6fb-d2a279e4e710 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:47.800516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e45962-3574-46d8-b6fb-d2a279e4e710 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:47.800977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e45962-3574-46d8-b6fb-d2a279e4e710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:47.825700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-26e45962-3574-46d8-b6fb-d2a279e4e710 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:47.827237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-26e45962-3574-46d8-b6fb-d2a279e4e710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:47.917628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 226/459] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:48.116659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab9bd40-7ce6-4275-b128-5a0ab3aa2fa6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.233005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c0d53e-6791-48a1-8cd9-39b71e0e6569 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.233205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c0d53e-6791-48a1-8cd9-39b71e0e6569 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.233769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c0d53e-6791-48a1-8cd9-39b71e0e6569 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.235820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 227/460] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:48.243389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.244002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.244471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.266011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:48.267508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:48.284526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab9bd40-7ce6-4275-b128-5a0ab3aa2fa6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:48.332032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09171902-f543-41e8-a483-f80624223080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:48.332942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09171902-f543-41e8-a483-f80624223080 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.333224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09171902-f543-41e8-a483-f80624223080 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.333497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09171902-f543-41e8-a483-f80624223080 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.340711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 228/461] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:48.341785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.356415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09171902-f543-41e8-a483-f80624223080 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:48.358221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09171902-f543-41e8-a483-f80624223080 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:48.374580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-09171902-f543-41e8-a483-f80624223080 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:48.377261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:48.378158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:48.379034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:48.379034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:48.379034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:48.381106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 234/462] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:48.382504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168ac6a-1b74-4884-8c6a-a564b41917d8 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.383039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168ac6a-1b74-4884-8c6a-a564b41917d8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.383497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168ac6a-1b74-4884-8c6a-a564b41917d8 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.385143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 235/463] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:48.391236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168ac6a-1b74-4884-8c6a-a564b41917d8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.431395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:48.479428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e200adfa-69a7-4a32-95f1-c95f0c298ab3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:48.525920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:48.526987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.527409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.527842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.534089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168ac6a-1b74-4884-8c6a-a564b41917d8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:48.541407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:48.543512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e05444fa-3828-4bbf-ac46-b1d8ab1a4656 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:48.563411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:48.564713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:48.565763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:48.565763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:48.565763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:48.568261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 229/464] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:48.569789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.571189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.571750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.574781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:48.575509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.575861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.576231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.577056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:48.577520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:48.579192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:48.595651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:48.597351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:48.603189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 236/465] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:48.603189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.687771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 230/466] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:48.688565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.691325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:48.731519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:48.746720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed0efe8-b734-4126-8d38-922b8f1aa170 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:48.800424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3fb1c-d6db-418f-a7a7-31d70e5cc193 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:48.801211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:48.802309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.803635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.804117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.805282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:48.805962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:48.807864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:48.827267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 237/467] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:48.828804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.842037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:48.843059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:48.843481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:48.844348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:48.845181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:48.847757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:48.881480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 231/468] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:48 2022] POST /identity/v3/users => generated 246 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:48.882730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:48.974883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:48.991420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197892c5-8dc8-44ab-93c3-7d252dde7529 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.038325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.039257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.039664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.040114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.049755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac93961-4f35-450d-9457-ca1eba5f1174 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.053939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.056460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:49.078358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:49.079560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:49.080400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:49.080400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:49.080400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:49.083930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 238/469] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:49.086457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:49.098988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:49.101259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.103087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.103719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.104228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.105224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.108184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.142703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 232/470] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:48 2022] POST /identity/v3/users => generated 248 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:49.144038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:49.253900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1468bfe0-b6b6-4a83-bcdc-afb0846a0513 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.296099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.297162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.297676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.298171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.312129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.314190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-44be8a81-db93-4d6a-9269-6f5f8a64aba8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:49.331581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:49.332621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:49.333656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:49.333656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:49.333656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:49.336105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 239/471] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:49.342580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f90ce3b-c99c-45f7-b3ce-a697f95691ea None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.343407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f90ce3b-c99c-45f7-b3ce-a697f95691ea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.343994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f90ce3b-c99c-45f7-b3ce-a697f95691ea None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.346122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 240/472] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:49 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:49.347682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f90ce3b-c99c-45f7-b3ce-a697f95691ea None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:49.349645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6fbf3-e216-4840-ba04-312b5f4e6432 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.376238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:49.404589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.405036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.405404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.405739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.406539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:49.407939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.409758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba451ee1-e018-4850-8538-848ad5395458 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.424933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:49.427039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 233/473] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:49.430573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.430573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.430573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.454593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:49.456048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:49.484105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f90ce3b-c99c-45f7-b3ce-a697f95691ea None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.528621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.529315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.529704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.529989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.530720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:49.531022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.532850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.548615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 234/474] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:49.549902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:49.550144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 241/475] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:49.559903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:49.675356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a810755f-61ad-44dd-bf6e-be063d8e08d3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.688694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:49.719301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.720304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.720777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.721260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.722150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.723814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.750299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 235/476] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:49 2022] POST /identity/v3/users => generated 248 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:49.813582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5cdc4a-f0ad-404e-8748-032e5b16aca3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:49.836048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:49.858007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:49.859024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:49.859555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:49.860046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:49.860931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:49.862782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:49.889025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 242/477] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:49 2022] POST /identity/v3/users => generated 249 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:49.912711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:50.052518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7238c4-da6c-4f04-9021-08a58c0063bb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:50.091384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:50.092013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:50.092291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:50.092564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:50.093265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:50.094682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:50.104099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.sql.core [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:21:50.104099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 08:21:50.104099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [parameters: {'id': 'dd388701574a49dc9310cc2a9046bfb8', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 08:21:50.104099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81404) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Oct 04 08:21:50.122866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return ref.to_dict() Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:21:50.123768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application yield resource Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise self.value Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application session.commit() Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:21:50.124636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:21:50.125473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:21:50.126963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:21:50.127767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [parameters: {'id': 'dd388701574a49dc9310cc2a9046bfb8', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:50.128544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 04 08:21:50.134502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:50.136595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 236/478] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Oct 4 08:21:49 2022] POST /identity/v3/roles => generated 144 bytes in 212 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:50.136734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:50.270127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f844bf-0093-4798-a16c-e310ba7c7019 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:50.306955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:50.307945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:50.308430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:50.308910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:50.320062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ea48a3a0-b2f7-4941-8529-1286b76e6755 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:50.326409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:50.327334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 04 08:21:50.328103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:50.330661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 243/479] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:21:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:50.336589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:50.402915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:50.482263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151abfc5-9d68-4ee9-a347-4e17e2e3b739 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:50.519479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:50.520093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:50.520373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:50.520643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:50.521311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:50.521613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:50.523087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a7c233-83d9-49bc-b59d-54419c56bbd1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:50.532580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 237/480] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:51.934924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5df59-b00a-44e4-88bd-907c64e3d3e6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:51.935525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5df59-b00a-44e4-88bd-907c64e3d3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:51.935525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5df59-b00a-44e4-88bd-907c64e3d3e6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:51.937108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 244/481] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:51.942218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:51.942690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:51.942909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:51.967585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:51.968776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:51.990143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f654e34-287a-483c-b2c4-33be6b5ff506 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:51.990903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f654e34-287a-483c-b2c4-33be6b5ff506 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:51.991710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f654e34-287a-483c-b2c4-33be6b5ff506 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:51.995023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 245/482] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:52.000114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461f5f0-d9e4-41e6-99df-dcdefe7daa11 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.000641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461f5f0-d9e4-41e6-99df-dcdefe7daa11 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.001172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461f5f0-d9e4-41e6-99df-dcdefe7daa11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.042865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2461f5f0-d9e4-41e6-99df-dcdefe7daa11 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:52.044061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2461f5f0-d9e4-41e6-99df-dcdefe7daa11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:52.079075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 238/483] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:52.100059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:52.176182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 246/484] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:52.421661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:52.444606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:52.580233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ecc6f-607e-42a8-9e61-3a5c3172d684 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.580861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ecc6f-607e-42a8-9e61-3a5c3172d684 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.581346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ecc6f-607e-42a8-9e61-3a5c3172d684 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.582702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b4f4c5-7d79-4ffe-9b7b-d7fd3ac56067 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:52.586635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 247/485] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:52.595200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.595749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.596215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.624089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:52.624735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.625004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.625271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.628428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:52.629849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:52.638278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:52.643577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:52.644462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:52.645241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:52.652668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 239/486] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:52.652668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:52.728497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 248/487] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:52.730043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:52.767964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:52.821846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60120db7-7156-4235-874c-7d54f6b84215 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:52.896906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:52.897542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.897815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.898082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.910348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:52.922183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:52.925225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0a0813-0ff4-44c4-bde7-42795e7c3492 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:52.932676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:52.933570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:52.934331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 240/488] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:52.941805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:52.991173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:52.993547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:52.994118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:52.994659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:52.995610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:52.996160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:52.997906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65751f7d-c380-4b50-a29a-b613c9993fc9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.007537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 249/489] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:21:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:53.010046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63125f6-a15b-4fa8-9098-8e6db49ecc3e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.010589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63125f6-a15b-4fa8-9098-8e6db49ecc3e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.011118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63125f6-a15b-4fa8-9098-8e6db49ecc3e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.012832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 250/490] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:53.013715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63125f6-a15b-4fa8-9098-8e6db49ecc3e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.086882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6ec96f-3569-4288-be6f-17756cd70d8a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.122694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.123394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.123674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.123946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.124594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:53.124918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.126245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626ebeee-04f4-4a73-93cb-687a10e8c62a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.128806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.136798 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 241/491] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:21:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:53.138068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.138345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.138645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.146771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63125f6-a15b-4fa8-9098-8e6db49ecc3e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.163270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:53.165542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:53.191149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.192340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.192796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.193245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.210634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.213751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:53.232803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:53.233901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:53.235608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:53.235608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:53.235608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:53.235608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 251/492] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:53.235608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.249559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.270627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 242/493] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:53.270627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.370660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7a018b-373a-4000-a457-f71e8484d013 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.396056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb6563b-2847-4a5a-8357-c817542264cf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.405543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.406166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.406452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.406971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.418422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:53.423690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:53.424558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:53.425562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 252/494] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:53.427043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.432887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.440963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.441853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.442133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.442402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.454480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.455952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4481eb69-bb7c-44be-83f8-d84af92fe65f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:53.476361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:53.479120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:53.480372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:53.480372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:53.480372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:53.482021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 243/495] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:53.483587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd752cd-fa00-4220-805d-bd59dac7fe4c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.483913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd752cd-fa00-4220-805d-bd59dac7fe4c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.484235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd752cd-fa00-4220-805d-bd59dac7fe4c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.485878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 244/496] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:53.486658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd752cd-fa00-4220-805d-bd59dac7fe4c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.511599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.547831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e19b92-99ae-4ffb-9f56-48713e762eea None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.581844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.582444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.582746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.583016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.583665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:53.584121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.585308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.598417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 253/497] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:53.599399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.612300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd752cd-fa00-4220-805d-bd59dac7fe4c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.646314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.646944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.647210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.647485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.648137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:53.648923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.650274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4b147-2b7b-48d9-8d5c-c477eb65dddb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.659401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 245/498] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:53.662291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.662704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.663114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.681841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:53.683056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:53.700959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915bef0-5222-42eb-b609-b4b666bd34bb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.717416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.727808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.737478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.738112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.738391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.738696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.739340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:53.739644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.741013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.750839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 246/499] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:21:53.751757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.757741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 254/500] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:53.757741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.862092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f019-9c4a-4bac-8828-69779848fcdb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.866154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966c2ca-a08e-40df-83f0-6e63d18a859e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:53.900692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.901320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.901595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.901869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.902460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:53.903208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:53.903579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:53.903957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:53.912893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:53.913765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:53.915137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:53.920148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:53.921212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:53.922210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 247/501] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:53.923471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:21:53.930613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:53.931616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 255/502] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:53.932665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:53.961990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:53.987579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.041430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a890be4d-af98-4a75-a67d-374e8efc6d92 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.045436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b85bce3-6745-485c-a798-ac8ce574259a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.079023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.079693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.079953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.080060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.080258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.080567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.080840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.081105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.089358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.090848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.092746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.094169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:21:54.101961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:54.103455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 256/503] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:54.105117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:54.113765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:54.114652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:54.115744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:54.115744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.115744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:54.115744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 248/504] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:54.115744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.204299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.216796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204f9cca-3875-403d-bf4f-05fecb7d3ab8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.223633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edd512d-5f6e-446d-a8a5-5700e7889e31 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.223764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.251873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.252476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.252756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.253022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.253677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:54.253972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.255369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.256811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.257410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.257684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.258655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.259449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:54.259761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.261156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.264130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 257/505] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:54.264928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.298389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 249/506] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:54.299175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.366771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64ce370-ff68-49fa-8424-e38b368eced2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.402204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.402865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.403135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.403413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.406007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330f15d4-ff8f-4057-be27-684a3727e21c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.413956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.415619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:54.429476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:54.431376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:54.432666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:54.432666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.432666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:54.432666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 258/507] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:54.432666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.446449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.447363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.447543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.447807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.448526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:54.448827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.450252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.454165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.464030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.472456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 250/508] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:54.473295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.532204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f973922c-46c8-4307-b638-92136784fe63 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.566677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.567276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.567548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.567813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.568454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:54.568750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.570103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.574735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335dc9aa-5156-48a9-a965-87fc609c70a3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.582910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 259/509] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:54.583749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.612535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.613155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.613511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.613690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.623872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.625245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:54.639751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:54.640621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:54.641504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:54.641504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:54.641504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:54.641504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 251/510] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:54.641713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.666528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.685605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f915c33f-18d4-4a50-9dad-5fc3be374282 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.693244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.727782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.728380 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.728658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.728928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.741969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f042c6-70ed-4bf5-9cef-9cc51eec7cb9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.775650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.776247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.776522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.776786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:54.777415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:54.777714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.779106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.792163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 252/511] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:54.793004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.796325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:54.798179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:54.876744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.897523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 260/512] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:54.898173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:54.898422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4a99db-5c4d-4039-abe9-2949c98d3280 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:54.914286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:54.936657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:54.937255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:54.937526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:54.937790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.002499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8ee3bc-55b9-4f1d-b5a2-e0c9f92d5afd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.008985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.010437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.037400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.038122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.038398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.038686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.045714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.047212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:21:55.059374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:55.060462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 261/513] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/users/cinder => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:55.061345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.101520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.110728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 253/514] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:55.111552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.126027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.162776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f14c5-468b-48b1-9e00-0e6272e90275 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.197365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.197958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.198226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.198490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.199161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:55.199459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.200826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.211709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97219299-a2b0-4fc5-96c9-f0e8d1719130 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.212549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 262/515] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:55.213326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.252015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.252714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.253061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.253489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.254287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.256206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.294580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 254/516] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:21:55 2022] POST /identity/v3/users => generated 247 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:21:55.294580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.318395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.322498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ac7d2-37c9-4a43-bda0-78c6f894eb65 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.362281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.362948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.363223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.363489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.377054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.378720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.379954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:55.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:55.403139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:55.404026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:55.404026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:55.404026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:55.404026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 263/517] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:55.405081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.425345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b3c4a-6b5e-47cd-9be3-7f4ba1958bc0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.462623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.463241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.463513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.463775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.464503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:55.464810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.466232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.487731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 255/518] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:55.488570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.549254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46bafd-acfc-4349-8e69-8cbfd2a5fb6d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.585393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.586591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.586874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.587139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.598190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.599706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:55.616666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:55.617617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:55.618446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:55.618446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:55.618446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:55.619387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86798131-b20d-45da-ac1b-373a59f0964f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.619474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 264/519] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:55.619630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.647757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.659462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.660070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.660369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.660682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.669788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.671440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:21:55.686618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:55.687554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:55.688452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 256/520] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/users/neutron => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:55.688985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.732253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba6d4-543c-41ec-ae5a-651ba1b66855 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.767127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.768069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.768341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.768608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.769255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:55.769549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.770953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.784163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 265/521] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:55.784939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:55.807679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f40179-3c45-449c-b2d5-61f684d94c2f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.844096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.845070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.845482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.845904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.856323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.902121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952c3307-1e83-4c8b-85bc-706ddb46bf55 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:55.918316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:55.939068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.940519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.946415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:55.947066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:55.947387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:55.947608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:55.948256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:55.948539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:55.949907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:55.966886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 266/522] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:55.966886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.047846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 257/523] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:56.051491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.084836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5898b52-cce6-4f67-9cc6-a25cbceb6b22 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.112221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.125635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.126246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.126524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.126827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.127577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:56.127887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.129531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.155679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.176949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 267/524] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Oct 4 08:21:55 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:56.177115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf4791-5f71-4da5-b07f-91acffa27cd1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.177213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.213434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.214074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.214351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.214660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.222976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.224657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:21:56.238232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:56.239184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:56.240074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 258/525] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:56.240522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.287613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65e93e-cd58-4d61-bfdc-2c03728fadb1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.327112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.327747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.328022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.328289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.336759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.338125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:21:56.349779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:56.350727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:56.351648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 268/526] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:56.352775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.359249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86857252-dac7-4b20-b7dc-f883ffb4c7f1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.401337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.401971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.402254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.402524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.403626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.415019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.416478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.423521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:56.433386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:56.434269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:56.435144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:56.435144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:56.435144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:56.435144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 259/527] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:56.435867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.471491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcbf981-ecaf-4def-824a-2e3a7579dc73 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.510771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.511765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.512251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.512726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.547665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328012e7-1073-425d-86f1-1289d70952c0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.583771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.584383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.584657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.584925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.585617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:56.585920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.589743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.596397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.597873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.614229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 260/528] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:56.614229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.647541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.683957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.718820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 269/529] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:21:56.718820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.734229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305491e8-fecc-4e73-843f-222c4f2a8425 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.769499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.770192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.770468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.770781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.771486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:56.771773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.773191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.790187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 261/530] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:56.790187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.828140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8168eae-78b5-448d-9bd8-53f145aa5bad None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.865526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.866480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.867069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.867402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.868538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:56.868823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.870194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.882521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 270/531] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:56.883985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:56.886708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.907493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7826e70d-0d3c-427f-9ba0-610c530ad2b8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:56.929647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:56.950709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:56.951248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:56.951703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:56.951980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:56.952722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:56.953025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:56.954517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:56.961765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 262/532] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:56.962489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.010906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a1db83-01cd-4266-ac85-576cde8a8056 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.052461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.053156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.053438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.053707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.069990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.072057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:57.088410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:57.089791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:57.090693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:57.090693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:57.090693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:57.090693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 271/533] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:57.090693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.119277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ce613d-0675-490a-90b4-797e34112606 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.160092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:57.174057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.175271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.175545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.175809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.176545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=bbc7631562d645dd9b7876a59de53a31) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:57.176843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.178282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.190624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 263/534] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:56 2022] GET /identity/v3/role_assignments?user.id=bbc7631562d645dd9b7876a59de53a31&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:57.190624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.221093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb4987-b55d-4c81-8588-87fa06a8e72e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.253896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:57.267056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.267831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.268183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.268527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.277599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.279241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:21:57.292268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:57.293156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:21:57.294296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 272/535] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:57.299349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.336823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4739a9-920d-47cf-9c88-1a599185ec4a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.401679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.402328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.402636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.402913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.403610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:57.403916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.405370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.434474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 264/536] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:57.439751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.470538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e5aad-5540-4157-882a-3986ee8273a8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.489926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:57.515572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.516311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.516590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.516856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.517544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:57.517841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.519443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.541025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 273/537] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:57.547162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.659280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bde191-bcb9-4da7-990f-ad1fb779356b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.672548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:57.676439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac68a6-6211-4ef0-b21b-94191bda6099 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:57.719907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.720526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.720807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.721104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.721816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=546c67695c1e447da68b91637aefe135) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:57.722125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.722403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:57.723273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.723639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a7747d-cfd5-42ae-9b4d-f6d6ec8d5746 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.723958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.724415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.731451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 274/538] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/role_assignments?user.id=546c67695c1e447da68b91637aefe135&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:57.732457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6b7ee-0f4e-4f85-a689-7b100d017e06 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.732744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6b7ee-0f4e-4f85-a689-7b100d017e06 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.733021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6b7ee-0f4e-4f85-a689-7b100d017e06 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.741841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 275/539] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:21:57 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:21:57.745807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:57.746142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:57.746368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:57.746682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:57.749412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:21:57.772725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:57.774265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:21:57.775852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:57.779994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 265/540] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:21:57.782192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:21:57.786610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:21:57.796495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:57.829036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:57.879681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 276/541] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:21:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:21:57.996709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c951e0-3e9d-477c-bf9a-a754d844a562 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:58.042094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:58.042741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:58.043022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:58.043300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:58.044002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:58.044306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:58.045744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:58.062030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 266/542] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:21:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:21:58.067956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:58.211774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb848f30-89af-4726-9a9a-a54b9bd99a75 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:58.249501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:58.250038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:58.250317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:58.250653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:58.251346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=3e66140a80f641f4afe94ed0452000f8) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:58.251808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:58.253494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:58.260675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 277/543] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:21:58 2022] GET /identity/v3/role_assignments?user.id=3e66140a80f641f4afe94ed0452000f8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:58.290005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:58.339956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:58.445766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2534b9-b1ec-465a-b623-f27a9873da97 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:58.487069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:58.487709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:58.487985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:58.488252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:58.507629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:21:58.513118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:58.513980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:58.515246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 267/544] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:21:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:58.522099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:58.562132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:58.651693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b26b623-f53d-4c57-b78d-34e7b787a002 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:58.697991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:58.698632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:58.698915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:58.699243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:58.699918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:58.700216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:58.701602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:58.711002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 278/545] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:21:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:58.720045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:58.817363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:58.834981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8820510d-039b-4447-ab31-83c08ee47e1f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:58.872582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:58.873219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:58.873494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:58.873763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:58.885192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:58.886691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:21:58.902750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:58.903740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:21:58.904555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:21:58.904555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:21:58.904555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:58.904555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 268/546] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:21:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:58.909840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:59.066146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:59.074765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519326e5-d0df-4170-8aa9-fb43fa8d564f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:59.126069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:59.126716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:59.126992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:59.127262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:59.127917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:59.128214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:59.129605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:59.146586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 279/547] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:21:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:59.152792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:59.327182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880ee33-38d7-4ee1-9357-5fb1f424a760 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:59.389119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:59.390167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:59.390665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:59.391130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:59.405378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:59.407210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:59.412136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:21:59.428043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:21:59.429463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:21:59.430989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:21:59.433501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 269/548] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:21:59 2022] GET /identity/v3/users/swift => generated 83 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:21:59.438880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:59.615422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2cd60a-a6e2-4a9b-bad6-6a4cd4872f93 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:59.660869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:59.661644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:59.662002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:59.662353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:21:59.663349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:21:59.663729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:21:59.666142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e45e12-1427-4d83-9214-470fa6ec8e2c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:21:59.690587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 280/549] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:21:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:21:59.698521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:21:59.844994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17451d9-90ce-4b93-b260-cc94cfd6ceb4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:21:59.899146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:21:59.900132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:21:59.900626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:21:59.901088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.003041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.007669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:00.010466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e7d48-616d-4590-8629-2700b18dcbbe None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:00.057840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629ec60-b3ef-45ef-abb0-b8b0d58984a9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.058189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629ec60-b3ef-45ef-abb0-b8b0d58984a9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.058389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629ec60-b3ef-45ef-abb0-b8b0d58984a9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.059959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 281/550] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:00.076480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d568ad-6070-4b0c-a987-1743a12f8be4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.076760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d568ad-6070-4b0c-a987-1743a12f8be4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.077032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d568ad-6070-4b0c-a987-1743a12f8be4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.107821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-98d568ad-6070-4b0c-a987-1743a12f8be4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:00.109123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-98d568ad-6070-4b0c-a987-1743a12f8be4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:00.130069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.176965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 270/551] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:21:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:00.179080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ffe4f-2e0d-45a4-bb74-427a56016ee8 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.179616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ffe4f-2e0d-45a4-bb74-427a56016ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.180638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ffe4f-2e0d-45a4-bb74-427a56016ee8 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.182776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 271/552] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:00.183848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ffe4f-2e0d-45a4-bb74-427a56016ee8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:00.240958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 282/553] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:00.243948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.244227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.244496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.278403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:00.280472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:00.353149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.367577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ffe4f-2e0d-45a4-bb74-427a56016ee8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:00.423322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 283/554] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:00.436916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:00.437602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.437881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.438143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.452749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:00.454186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:00.473455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:00.474675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:00.475667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:00.475667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:00.475667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:00.475667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 272/555] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:00.483424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:00.559027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:00.587094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.625508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da8a10d-2227-4935-a9be-cd7beba2c637 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:00.673202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:00.673872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.674136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.674324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.675001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:00.675288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:00.676769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:00.685048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c171ad9a-34a3-4d5a-92f0-86713a155b67 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:00.703201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 284/556] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:00.715637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:00.734952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:00.735596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.735871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.736139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.756039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-70fb1046-d1d1-4a61-a85a-8c613999ab66 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:00.763249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:00.764410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:00.765282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:00.767655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 273/557] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:00.774044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54361209-a307-4bd3-8589-6c9bbb65519c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.774832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54361209-a307-4bd3-8589-6c9bbb65519c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.775324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54361209-a307-4bd3-8589-6c9bbb65519c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.777304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 274/558] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:00.778201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54361209-a307-4bd3-8589-6c9bbb65519c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:00.796292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.835526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ebedcb-4849-4742-96be-e5c08941159d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:00.868817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:00.875568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:00.876223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.876503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.884842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:00.886347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-eeb71766-5714-4b00-809f-7d2e60322f3b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:00.899310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:00.900271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:00.901134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 285/559] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:00.901913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.902082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.902359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.909389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54361209-a307-4bd3-8589-6c9bbb65519c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:00.921390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:00.922608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:00.949796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:00.950731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:00.951183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:00.951613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:00.952451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:00.952761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:00.954318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:00.969070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 275/560] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:00.969070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.002354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 286/561] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:01.003413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.055293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.090070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811eb686-64e0-41e0-9959-cb09dbb2a089 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.115791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02202e06-44c1-4681-b675-563eaf1b3679 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.124470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.125099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.125377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.125645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.136726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:01.142660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:01.144121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:01.144977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:01.148108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 276/562] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:01.149595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.153708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.170587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.171290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.171657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.172017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.172800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:01.173181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.174908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f4604-3d87-40f2-a31c-9d3959a7e4fe None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.190266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 287/563] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:01.191119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f02c181-0114-4cf0-8687-6e03cef5450d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.191468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f02c181-0114-4cf0-8687-6e03cef5450d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.191809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f02c181-0114-4cf0-8687-6e03cef5450d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.193269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 288/564] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:01.193914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f02c181-0114-4cf0-8687-6e03cef5450d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.285702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f2dd6b-e435-4f75-98b8-b043232e5cd7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.318757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.324186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f02c181-0114-4cf0-8687-6e03cef5450d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.324633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.325265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.325542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.325814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.336718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.338164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:01.352866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:01.354426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:01.355372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:01.355372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:01.355372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:01.355372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 277/565] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:01.355372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.363119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.363724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.364002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.364276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.364949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:01.365254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.366665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ec129-0a87-49e7-a364-e874fd7bf94c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.378017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 289/566] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:01.379111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.379399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.379675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.399981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:01.401189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:01.443823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.471292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae17865-600e-4a10-b3e0-b304d64e0f5b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.476637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 290/567] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:01.478097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.519327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.519977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.520187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.520458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.521182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=5cc208be4d444563929a03634cfddee7) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:01.521484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.523412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.538685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 278/568] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/role_assignments?user.id=5cc208be4d444563929a03634cfddee7&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:01.538685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.615233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.628146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a309694-3ec5-4af4-8eec-ddc64d197526 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.669664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.670407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.670816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.671124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.684811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:01.691095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:01.692321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:01.693249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:01.694649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 291/569] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:01.698167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.707006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94a2454-7e42-4d75-8020-ee4b905b81b5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.746511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.755619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.756235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.756508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.756774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.757467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:01.757770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.760218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.775131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 279/570] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:01.776268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.827579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c26d5a-3d26-4068-8c00-b94cc4163e84 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.865909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.866629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.867096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.867415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.877992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.879480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.884757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1ee27c-28b9-43a2-841e-18f78b6404d6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:01.895039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:01.895953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:01.897037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:01.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:01.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:01.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:01.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 292/571] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:01.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.931326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:01.931979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:01.932282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:01.932578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:01.933301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:01.933726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:01.935427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:01.951821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 280/572] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:01.951821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:01.996846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.013951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4859196-5248-431c-8fbb-8f7b59a52a97 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.052000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.052799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.053080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.053406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.066929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:02.071726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f6e844-59ca-4287-815c-876ea06f5d0e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:02.074777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.075720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:02.076621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 293/573] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.077341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.112395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.113263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.113706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.113975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.122771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.124199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:02.136336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.143398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 281/574] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:01 2022] GET /identity/v3/users/glance => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.144381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.181386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.199252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113f753-bd84-4513-87bc-15afab742e35 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.234924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.235682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.235958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.236221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.236873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:02.237156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.238517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.252205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 294/575] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:02.257379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.259486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47f20ec-7b73-4ee8-81eb-f42e783a6328 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.266206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.297948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.298594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.298871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.299137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.311271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.312713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:02.328954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.329945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:02.330847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:02.330847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.330847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:02.334661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 282/576] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.335520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.369044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48998e0e-f729-4e0f-8126-c0fc152ab714 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.406514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.407542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.408068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.408337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.409002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:02.409303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.410695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.420413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.420900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 295/577] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:02.421700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.444721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020599b-c79c-4b91-a316-e0f6f47da8e5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.490652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.491425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.491768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.493466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.494062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.494863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:02.495175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.497046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.518142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 283/578] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:02.523461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.533548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ba6bf-d054-45e7-bc75-76c215a2fe34 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.567999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.568618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.568891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.569157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.577564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.579045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:02.590285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.591239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 296/579] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.592169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.640986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd4913-69f1-44bf-b427-d61fc75a3d45 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.677644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.678262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.678520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.678811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.679445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:02.679740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.681649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.690268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.700461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 284/580] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:02.700461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.704961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1670235-60f9-4d10-9773-06d9f0cbbef4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.737260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.746667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.747322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.747606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.747896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.761225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.762866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:02.778445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.779736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:02.780653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:02.780653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.780653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:02.780653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 297/581] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.780906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.820123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2676b3db-da02-4955-a7a3-cdb7ab326e74 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.855582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.856207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.856480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.856748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.868947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.870389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:02.885454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:02.886638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:02.887663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:02.887663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:02.887663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:02.887860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 285/582] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:02.888631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:02.903377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e5d7-ca9e-42d4-a79a-73ab2c9faa04 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:02.942538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:02.949818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:02.950487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:02.950895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:02.951811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:02.952565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:02.952908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:02.954896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:02.979331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 298/583] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:02.980755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.010065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f13ca4-ad19-4adb-bf82-905864a55bc6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.036464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.049825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.050539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.050916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.051286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.061838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.063442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:03.075793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:03.076944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:03.078098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:03.079050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 286/584] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/users/cinder => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:03.079702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.109080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c821406e-0785-46e0-8d7d-fa7e2ad5f609 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.157881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.158576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.158872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.159149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.159793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:03.160124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.161618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.175206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 299/585] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:03.176054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.204026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97941c8-8044-4bb2-a9af-14bc31c17ab0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.221790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.256372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.257338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.257795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.258249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.259327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:03.259829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.261896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.286078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 287/586] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:03.286078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.308910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.313247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b954f0-a656-4758-bdd2-09b1c5022ec9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.355594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.356234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.356511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.356780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.368864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.370377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:03.388552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:03.394750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:03.395694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:03.395694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:03.395694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:03.395694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 300/587] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:03.395694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.419302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb341b63-a113-470a-8614-abd43fc3873c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.466828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.467753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.468120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.468311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.469407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:03.469951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.471333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.491854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 288/588] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:03.491854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.532985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.537388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05aa23-ea6f-454f-b2d5-5b01a4062741 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.581456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.582134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.582407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.582716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.592594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.594247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:03.607684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:03.608972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 301/589] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/users/neutron => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:03.610510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.625137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb3472b-8b73-4b3f-837b-f42f923820a4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.627368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.668287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.669086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.669477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.669688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.677877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.681394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:03.698063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:03.699796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 289/590] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:03.700911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.756695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d03d6e-5a40-4b08-8916-edc0d78ef398 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.801279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.802063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.802493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.802918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.803884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:03.804393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.807657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.814497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.820487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 302/591] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:03.822670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.825659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa59b71-352a-413d-9fa6-0b9ea5e0639e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:03.865235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:03.866222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:03.866693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:03.867392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:03.883901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:03.886432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:03.910162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:03.911320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:03.912323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:03.912323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:03.912323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:03.912323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 290/592] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:03.912910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:03.925659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:03.966559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5988a0-0353-47f8-b8ef-7f853f1b3e20 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.005727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.006706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.007162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.007514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.009276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:04.009657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.012101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768afd7-276c-44ac-9d5f-bed16d1ce8c0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.033061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 303/593] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:04.034318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8e50d-6b8a-44b0-90a6-ffca8ff28e7e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.034629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8e50d-6b8a-44b0-90a6-ffca8ff28e7e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.034901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8e50d-6b8a-44b0-90a6-ffca8ff28e7e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.038846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 304/594] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:04.039004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5671453-63d3-4ae5-b27b-7465cfb332ed None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.039129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8e50d-6b8a-44b0-90a6-ffca8ff28e7e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.074889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.075480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.075756 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.076025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.076721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:04.077022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.078436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.091867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 291/595] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:04.092696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.095562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.153779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8e50d-6b8a-44b0-90a6-ffca8ff28e7e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.183673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.191961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.192644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.192919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.193191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.199932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.201373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:04.212731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:04.213739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 305/596] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:04.214716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.222772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75ba3f7-b58b-4a81-83fa-7621757552e1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.257699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.258317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.258612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.259051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.260481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:04.260808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.262170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d897f2f-aeb9-4e8d-aa60-6a3327e2c43b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.278826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 292/597] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:04.278826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.278826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.278826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.296267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:04.297485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:04.318308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2abc169-4cb8-4ad5-84b7-95acf99e3829 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.356031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.361738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5135116-e418-4026-8331-95afbd43891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.362489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5135116-e418-4026-8331-95afbd43891e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.362803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5135116-e418-4026-8331-95afbd43891e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.363081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5135116-e418-4026-8331-95afbd43891e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.367456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 293/598] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:04.368979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.387768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5135116-e418-4026-8331-95afbd43891e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.389261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5135116-e418-4026-8331-95afbd43891e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a5135116-e418-4026-8331-95afbd43891e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:04.404788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:04.405775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:04.406785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:04.406785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:04.406785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:04.406785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 306/599] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:04.406785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5135116-e418-4026-8331-95afbd43891e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.457997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.505144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaefdf5-90db-4264-8672-48ae43173952 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.534010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5135116-e418-4026-8331-95afbd43891e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.555729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.556996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.557475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.557779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/bbc7631562d645dd9b7876a59de53a31/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.570367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.571536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.571831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.572076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.572764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:04.573064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.574513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.574643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=bbc7631562d645dd9b7876a59de53a31, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.576090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.587779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 307/600] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:04.588806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.597385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 294/601] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:04 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/bbc7631562d645dd9b7876a59de53a31/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:04.598624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.656882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.706429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:04.710105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13bb95a-79f7-476a-9ccd-6eaafa3b7bd2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.731790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e3730-0745-4298-83b6-981018e4f7ad None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.749810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.750762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.751044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.751312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.761220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.762780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.774408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.775104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.775498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.775767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.776644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:04.776947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:04.777577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:04.778463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:04.779791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:04.780462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.783008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 308/602] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:04.784478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.807943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 295/603] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:04.810107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:04.918427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9f0ec-a73d-406c-97e4-c54239d4feb3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:04.956974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:04.957667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:04.958028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:04.958363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:04.974955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=02d91fccc1c8400392fbbc34416f552b, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:04.976443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:04.998818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 309/604] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:04 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:05.000088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.004896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e457ec-9556-4c7c-95a0-8176f2cbe4da None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.006712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.017225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.152130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.153683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.155683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.155683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.170087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdea4e-d038-4b6c-b5c6-5b39f11c471f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.182525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:05.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:05.201395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 296/605] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 394 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:05.207915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.251002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.252564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.253076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.253535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.254417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:05.254922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:05.259350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:05.272761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 310/606] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:05.274117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.376059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e95d97-bbb1-4323-9bc3-76a62f83d3c8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.435395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.459941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.461576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.462596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.463097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.476563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:05.478197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:05.495151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:05.496095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:05.496996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:05.497419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 297/607] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:05.498117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.511729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f935a0-34ab-440b-8f19-2ab8a16a3641 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.532714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.559212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.560165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.560612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.561022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.561835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:05.562385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:05.564104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:05.576693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 311/608] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:05.577779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.638687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b775f7-8157-492a-ba67-51415c553089 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.686882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.687914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.688438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.689128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/546c67695c1e447da68b91637aefe135/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.709446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da542e-3e58-4ccb-b99e-22e2eead9c28 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:05.709720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=546c67695c1e447da68b91637aefe135, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:05.711374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:05.736584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 298/609] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:05 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/546c67695c1e447da68b91637aefe135/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:05.737984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.757486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.769727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:05.770343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:05.771436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:05.771546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:05.773231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:05.773543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:05.777182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:05.802699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 312/610] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:05.810503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:05.914261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:05.980228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec9f23a-aefc-4122-9d5a-310d7f386f05 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.012790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e87d07-0224-4806-8020-cb0e6d0a4cc6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.023427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.024202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.024410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.024599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.039207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.042210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.065052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.065710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.065987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.066259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/3e66140a80f641f4afe94ed0452000f8/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:06.067029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:06.067965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:06.068832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:06.068832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:06.068832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:06.071338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 299/611] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:06.077847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:06.092166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=3e66140a80f641f4afe94ed0452000f8, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.093888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.116863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 313/612] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:05 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/3e66140a80f641f4afe94ed0452000f8/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:06.217738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e143d0-464c-43f7-a7fe-8684216b5f5b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.244204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:06.256351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.257391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.257900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.258396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.259261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:06.259852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.261418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.276421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 300/613] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:06.282379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:06.330895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:06.394991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77921eab-d3a4-4e48-801e-59b2c02e3432 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.434851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.436104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.436627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.437261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.448987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.451586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:06.466521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:06.467782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:06.469028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:06.471858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 314/614] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:06.478950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:06.611855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd0985b-89fa-4cc5-b679-15803ba52954 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.653980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.654578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.654976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.655216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.656048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:06.656360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.657995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.687002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 301/615] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:22:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:06.692828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:06.870858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbbd69f-0f77-44a4-aa5c-7ad153b7fa56 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:06.872622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:06.912069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:06.912725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:06.913006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:06.913275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:06.927049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:06.928577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:06.949756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:06.950784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:06.952010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:06.952010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:06.952010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:06.952010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 315/616] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:06.959682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:07.045067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:07.119893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046b6a6d-164e-45f5-bf40-cce476cb3f78 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:07.183299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:07.183965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.184239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.184531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.185214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:07.185512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:07.187136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc01b7a3-c33b-485a-a0a2-5d9776619869 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:07.206065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 302/617] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:07.214197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:07.228721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab5504-ed3a-4f35-8531-542b1f73f6b0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.229428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab5504-ed3a-4f35-8531-542b1f73f6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.230008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ab5504-ed3a-4f35-8531-542b1f73f6b0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.233427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 303/618] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:07.240917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.241568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.242034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.278166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:07.280984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:07.423170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd39070b-009e-4a73-991f-bdfe5dc134db None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:07.430732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 304/619] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:07.481491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:07.482511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.483005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.483466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.494177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:07.495930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7519d82f-73dd-4478-9ed5-7ad8965a577c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:07.518748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:07.519694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:07.520560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:07.523143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 316/620] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:07.528777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:07.576790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eec8a8-831c-4e9d-939c-d5587eee7663 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.577110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eec8a8-831c-4e9d-939c-d5587eee7663 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.577430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eec8a8-831c-4e9d-939c-d5587eee7663 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.578915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 317/621] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:07.579793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:07.586320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.587215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.587701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.629307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:07.630525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:07.658903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:07.683036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398ed69e-b319-4fb7-a445-57965131fe53 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:07.719647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:07.720417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.720693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.720960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.722461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:07.722793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:07.724270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:07.738273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 305/622] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:07.744965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:07.786812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 318/623] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:22:07.864341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:07.879914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45542895-8210-49da-a45b-8d262f8f943f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:07.889143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:07.916477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:07.917141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:07.917475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:07.917684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/5cc208be4d444563929a03634cfddee7/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:07.933319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=5cc208be4d444563929a03634cfddee7, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:07.934998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:07.955561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 306/624] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:07 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/5cc208be4d444563929a03634cfddee7/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:22:08.062104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa5a52-dbe1-4c59-b0b5-76236ba7c182 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.105971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.106623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.107099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.107374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.119611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:08.124770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:08.125760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:08.127150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 319/625] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:08.135567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.140174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.148461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:08.306963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73db93f-77ad-44f1-a7e8-db1f8b02f76d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.335294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5079c201-8920-4180-b6ad-083d058ad1cf None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.349547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.351013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.351493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.351959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.353880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:08.354577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:08.356279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:08.369523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 307/626] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:08.372603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.380912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:08.408543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.409483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.409937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.410518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.431924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:08.439943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:08.441124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:08.442184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:08.445652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 320/627] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:08.452290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.561936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:08.608050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dc803d-54ce-4b9b-b657-7d7b2aff8624 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.631866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f523ce-9b10-449d-bd57-286827821632 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.667253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.667902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.668178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.668447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.693219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:08.694728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:08.716569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.717183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.717457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.717720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.718384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:08.718707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:08.720084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:08.723708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:08.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:08.725609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:08.725609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:08.725609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:08.725609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 308/628] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:08.738764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 321/629] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:08.740216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.740405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.844469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:08.882396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29427c7-8e44-439a-839b-c79dd1cc71f1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.910829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:08.911448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1957e7e3-59ba-440c-a706-e0fb73dc0832 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:08.921465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.922476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.923067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.923562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.924486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:08.924992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:08.926631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb672884-b78c-4d90-a27b-8cd78a7f65e1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:08.941580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 322/630] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:08.943293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd91d910-76a7-49ff-8dd2-fa5a1c4e306b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.943786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd91d910-76a7-49ff-8dd2-fa5a1c4e306b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.944348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd91d910-76a7-49ff-8dd2-fa5a1c4e306b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.950908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 323/631] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:08 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:08.952082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd91d910-76a7-49ff-8dd2-fa5a1c4e306b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:08.957444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:08.958071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:08.958343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:08.958635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:08.973329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:08.976044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-97bfd6ba-1683-4728-8e39-74c2e6da7a26 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:08.992029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:08.994005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:08.995148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:08.995148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:08.995148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:08.998383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 309/632] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:08.999807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76669c6-26ae-46d9-9631-bb92c0d1fb66 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.000082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76669c6-26ae-46d9-9631-bb92c0d1fb66 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.000350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76669c6-26ae-46d9-9631-bb92c0d1fb66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.019671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f76669c6-26ae-46d9-9631-bb92c0d1fb66 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:09.020848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f76669c6-26ae-46d9-9631-bb92c0d1fb66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:09.073593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd91d910-76a7-49ff-8dd2-fa5a1c4e306b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.095993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 310/633] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:09.097166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f2c142-6ff6-4c1c-bb96-fc5f715f76cd None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.097447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f2c142-6ff6-4c1c-bb96-fc5f715f76cd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.097715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f2c142-6ff6-4c1c-bb96-fc5f715f76cd None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.099158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 311/634] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:09.104176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f2c142-6ff6-4c1c-bb96-fc5f715f76cd None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.119720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.120374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.120650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.120914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.131411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.133199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d5f4420a-8d58-4ff4-81bf-454bd2ca5198 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:09.147241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:09.148234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 324/635] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.154031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.154749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.173083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.173482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:09.174771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:09.231307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.261093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f2c142-6ff6-4c1c-bb96-fc5f715f76cd None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.266917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 325/636] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:09.267901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.300949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.302312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.302790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.303230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.304062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:09.304532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.306494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.320329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 312/637] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:09.327962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.440884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e5e63-e283-4247-9ea0-7f8c71f5b95b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.461723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.466280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d1b6a-201a-43f7-b2ba-ea89b410e528 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.487413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.488187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.488565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.488934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.489831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:09.490237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.491882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.511040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.511919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.512343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.512757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.513335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 326/638] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:09.514354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.522413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.523888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:09.541534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:09.542763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:09.543835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:09.546536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 313/639] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:09.547581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.571262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.657290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd798c-3621-44fd-a677-0d7c6e6a34c6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.671198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013410d9-afc0-4b50-a8d1-2cb7ac722e54 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.698706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.699265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.699541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.699816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.709605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.710291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.710666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.711005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.711396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:09.716515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:09.717658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:09.718779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:09.722085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 327/640] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:09.722987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.745713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.794698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.796287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.822563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:09.829499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03884f5c-5835-46ec-81af-94019a14085c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:09.867686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:09.868949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:09.869411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:09.869857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:09.870803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:09.871171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:09.872698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:09.895012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 328/641] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:09.895906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.909009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 314/642] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:09.909912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:09.976861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.009964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0270b91-c5ed-4786-a018-9ab7cb65bf56 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.023526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb8980-58e8-4068-8926-fe069162f007 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.051429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.052152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.052748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.052888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.060085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.060840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.061217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.061577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.062331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:10.062753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.064072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:10.064327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:10.069094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.070220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:10.071309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:10.074866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 329/643] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.075694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 315/644] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:10.076128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.076771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.094953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.192254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dbb252-cafc-4fcc-86ad-031b559f08c0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.203093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffae59b3-25e9-42e4-b091-55ef243e3fb9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.214820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.231910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.232465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.232878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.233071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.240978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.241578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.241851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.242123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.244946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.246591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:10.265272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.266407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:10.267518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:10.267518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.267518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:10.267518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 316/645] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.267946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.322412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.324127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.341021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.392059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f1a051-cbe9-4e5c-9f8d-09184daa04d0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.436072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.436716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.437038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.437309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.438052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:10.438353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.439890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.447043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 330/646] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:10.447043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.454358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 317/647] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:10.459033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.493973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.553853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d45afcb-029a-46f4-85c5-5f7cbe375779 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.563566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e315c24-49ef-4634-b1c2-55a337d585c8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.567016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.589588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ff726-e999-44ea-823d-67201b25525d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.590214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ff726-e999-44ea-823d-67201b25525d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.590488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ff726-e999-44ea-823d-67201b25525d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.590783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294ff726-e999-44ea-823d-67201b25525d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.597418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.598017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.598295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.598588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.599223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:10.599522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.600471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ff726-e999-44ea-823d-67201b25525d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.600875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.601892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294ff726-e999-44ea-823d-67201b25525d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.612598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 318/648] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:10.613401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-294ff726-e999-44ea-823d-67201b25525d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:10.616561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.618385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:10.619784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:10.619784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.619784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:10.619784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 331/649] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.620020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ff726-e999-44ea-823d-67201b25525d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.718562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.719650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294ff726-e999-44ea-823d-67201b25525d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.726215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12d21c-0724-46d3-b58b-b852506e9628 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.754606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.755145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.755424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.755687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.762252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.762914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.763626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.763901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.766782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.768220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.775734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.777543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-50c2bf74-ef70-406a-be4d-a40111601ce5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:10.782654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.783841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:10.785390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:10.785390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.785390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:10.785390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 332/650] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.785390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d106-8eb1-4dda-a425-0fe7cbe7a653 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.785673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d106-8eb1-4dda-a425-0fe7cbe7a653 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.785748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d106-8eb1-4dda-a425-0fe7cbe7a653 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.787247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 333/651] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:10.787627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304d106-8eb1-4dda-a425-0fe7cbe7a653 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:10.796839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.798963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:10.800443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:10.800443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:10.800443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:10.800443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 319/652] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.800443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.811329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.899032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304d106-8eb1-4dda-a425-0fe7cbe7a653 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.902879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82381198-24fe-444c-a8a3-f5eb455cd2dd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:10.933336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.933960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.934233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.934499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.937609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:10.938206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.938483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.938785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.939453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:10.939759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.941049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:10.941147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.942491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:10.953745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 320/653] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:10.954567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8b9cc790-e5fd-42fa-a793-6a38b8d9afcf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:10.954679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:10.955575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:10.956538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 334/654] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:10.957061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:10.957061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:10.957675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:10.968229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:10.982374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:10.986070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:11.054164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 335/655] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:11.055038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.064645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848779e-f74e-4931-9b9a-c52d43853210 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.071142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.099854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.100459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.100731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.100995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.101627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.101923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.103934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.117016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 321/656] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.118817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.187331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a1afa9-d035-410b-8e62-be12fc324566 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.204028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.230114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.230834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.231109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.231372 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.232017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.232318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.233709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.246884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 336/657] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.248122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.255246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21274727-bbee-4cb7-88bb-96e4c8e3425f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.300304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.300987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.301260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.301560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.315202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.316587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.324810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:11.328312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:11.329262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:11.330192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 322/658] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:11.331709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.367050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702bb6c-0364-438d-841e-d13d2fa2f669 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.407217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.407822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.408094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.408356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.409135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.409436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.411077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.424417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 337/659] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.425312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.443586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a38abf6-1b5e-46d7-a2e5-3d846cc351c8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.470532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.480581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.481324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.481638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.481908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.490059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.491622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:11.503465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:11.504512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:11.505571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 323/660] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/users/neutron => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:11.506509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.539160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2091023-7cba-4cfd-9772-26cb998ae0ca None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.566348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.576268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.576896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.577197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.577439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.584145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.585587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:11.597427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:11.599021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 338/661] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:11.600022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.619702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e1248-9278-4dbc-a21b-8ef7ce5245c0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.655947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.656576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.656849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.657119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.669175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:11.674386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:11.675512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:11.676507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 324/662] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:11.677304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.709728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61811e9b-24a1-4809-9936-a2cd760bdfdc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.726146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.746676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.747303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.747566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.747837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.748557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.748885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.750288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.776386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 339/663] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.777818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.785395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f15a7f-08be-487e-82f4-deee71eb8c55 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.818958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.819407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.819681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.819946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.820671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=bbc7631562d645dd9b7876a59de53a31) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.821554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.823459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.825053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.829725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 325/664] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/role_assignments?user.id=bbc7631562d645dd9b7876a59de53a31&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.831071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:11.921164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b381d4-17b6-4651-9686-2a289ad20556 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.959209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba0535-cde9-459b-9a66-df7730154e71 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:11.964485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:11.965252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:11.965662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:11.966084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:11.967233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:11.967604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:11.969693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:11.972893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:11.990951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 340/665] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:11.991901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.010043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.010693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.010969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.011243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.012527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:12.012527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.013665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.027391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 326/666] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:12.028258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.123290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b82796-5ec7-46ad-8b1c-85b9baa82fc0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.125410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.146961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fd71b4-ad29-404a-b114-4740ee31f5bd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.175291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.175908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.176185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.176456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.177127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:12.177432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.178840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.188123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 341/667] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:12.189012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.195568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.197035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.197452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.197890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.247452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.286995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.288682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.298213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348b454b-4f33-47d5-9025-d3609254c0a9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.336259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.337415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.337694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.337967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.372842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.398119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 327/668] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:12.398119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.419234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.420756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.481801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.517935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d15aa-d3a0-40b4-86f5-b4ea7d2caca9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.527326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 342/669] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:12.528360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.559425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.560150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.560428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.560733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.561422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:12.561726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.563316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.578597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 328/670] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:12.578597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.604209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.655486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4ca210-b994-4c56-b3ba-760bb9f37da5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.700728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.701531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.701819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.702089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.715183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.716690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.723172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad881f-f3c6-4f75-919f-12364f612a99 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:12.734844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:12.736083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:12.737093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:12.737093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.737093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:12.739246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 343/671] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:12.739246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.765360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.766829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.766829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.766829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.769693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.781262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.782927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:12.803183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:12.804195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:12.805535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:12.805535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.805535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:12.805535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 329/672] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:12.806000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.873056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a056c7d-ace0-46c6-9579-29208b172d72 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.893827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:12.920534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.921198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.921473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.921736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.934247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.935825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:12.948845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8815b27-1249-49df-8c6d-a6fada61bf9a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:12.951879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:12.952794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:12.954070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:12.954070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:12.954070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:12.954070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 344/673] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:12.954070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:12.985808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:12.986457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:12.986761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:12.987196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:12.988160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:12.988607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:12.990095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.003884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 330/674] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.004797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.059662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.099416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef97c39-c18d-42e0-8f2e-92a3672b011f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.127112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ceece3-543d-4c93-80be-13dae8a1f829 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.142506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.143417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.143417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.143871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.146574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:13.146574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.146574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.167327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 345/675] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.168261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.175503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.176035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.176310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.176576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.177228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:13.177521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.179043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.192553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 331/676] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.193448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.205200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.301566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035c0ec-57fe-4bda-83b9-0551aa242105 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.323779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b2a4b-a29d-4bee-a1dc-324d42851072 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.345734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.346208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.346527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.346826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.354432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.355972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.365517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.366361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.366800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.367208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:13.368360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:13.370375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 346/677] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:13.372277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.381222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.383141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:13.397398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:13.398460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:13.400136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:13.407684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 332/678] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:13.409118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.470568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.487232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f4deb1-f35c-40e9-8c2d-0b9f32c2d1a4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.523074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.523864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.524929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.525364 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.533179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.534805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.540983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed5e4f-5d28-4c45-bf0b-a6e9b1c9d40a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:13.546790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:13.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:13.548930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:13.550613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 347/679] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:13.551237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.595159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.597777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.598227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.598747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.599680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:13.600613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.602804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.629034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 333/680] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.630453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.665495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.670811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b386ff9e-3e0e-4ecd-adc2-5cf5af902cbc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.721512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.722353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.723085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.723501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.724350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:13.725046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.727616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.750701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 348/681] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.752018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.766938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.770138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa075dc-268d-4d48-8e16-82a63560c494 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.814494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.815447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.815854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.816256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.817212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:13.817846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:13.819711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:13.840643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 334/682] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:13.840643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:13.889663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d93eaca-c29f-41f4-9bde-b6bbabac06b7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:13.927378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:13.928177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:13.928587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:13.928965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:13.952669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:13.975473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043bc039-6656-425c-b353-1cb8e6d0d809 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.001580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.003298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96af591-a39a-4c22-861f-2958a35dd554 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.014573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:14.023318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.024715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.025266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.025676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.026885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=546c67695c1e447da68b91637aefe135) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:14.027351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.028908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.043615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 335/683] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/role_assignments?user.id=546c67695c1e447da68b91637aefe135&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:14.045858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.187974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 349/684] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:14.189456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd88fb1-f437-41b1-92fe-52e18da11751 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.189870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd88fb1-f437-41b1-92fe-52e18da11751 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.190252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd88fb1-f437-41b1-92fe-52e18da11751 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.191956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 350/685] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:14.197122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd88fb1-f437-41b1-92fe-52e18da11751 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.247149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef683e12-69f1-4d28-b3b6-52fd37c491f8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.304215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.305066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.306041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.306451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.308191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=3e66140a80f641f4afe94ed0452000f8) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:14.308670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.310348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135e6d0-7136-4211-a687-85d8d7480f37 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.316813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 336/686] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/role_assignments?user.id=3e66140a80f641f4afe94ed0452000f8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:14.318669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.323000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.323402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.327896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd88fb1-f437-41b1-92fe-52e18da11751 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.341864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:14.359737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:14.361542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:14.375243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:14.378859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.380596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.381160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.381726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.404189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.405693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.459688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 337/687] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:14.462137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:14.463327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:14.464427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:14.464427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:14.464427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:14.464427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 351/688] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:14.475799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.609048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f08bc2-eba6-42e5-b0e6-d6097af3117a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.652782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.653416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.653684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.653949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.654618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:14.654922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.656283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.670213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 338/689] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:14.675667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.791862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.794824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:14.805100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb33418-b594-48c4-9517-977dffd918d5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.854396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.855303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.855722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.856135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:14.875959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:14.877698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:14.902747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:14.903860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:14.904836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:14.906859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 352/690] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:14.919717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:14.937495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba3960-22df-4ad4-8f14-4ac7d19784f9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:14.959081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:14.980899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:14.981789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:14.982207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:14.982646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.018574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:15.018574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:15.045300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:15.046332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:15.048362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:15.048362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:15.048362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:15.048362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 339/691] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:15.054589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:15.143202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf582a3-20aa-43d2-a9f3-01b6b7d7eb50 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:15.171290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:15.205896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:15.206637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.206812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.207095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.207801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:15.208102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:15.209972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:15.227797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 353/692] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:15.236599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:15.284320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ed850b-cf49-4cb8-8e0f-3740c9c97ab6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:15.350252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:15.350889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.351163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.351615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.352436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:15.352816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:15.354996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:15.391995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 340/693] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:15.398174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:15.406828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:15.457651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba51b3c-c235-44b2-b380-744ba7bbe8ff None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:15.552614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:15.554072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.554874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.555326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.556838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=5cc208be4d444563929a03634cfddee7) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:15.557330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:15.558974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18867e67-5f54-465f-b507-131a8f184c78 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:15.567040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 354/694] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:15 2022] GET /identity/v3/role_assignments?user.id=5cc208be4d444563929a03634cfddee7&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:15.601067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadc1ad8-a73a-485f-86f9-3bf321f47823 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.601456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadc1ad8-a73a-485f-86f9-3bf321f47823 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.601739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadc1ad8-a73a-485f-86f9-3bf321f47823 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.604512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 355/695] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:15.611127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.611656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.612079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.636078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:15.638608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:15.640686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea34bf-2b27-4c54-92e4-71a05b7bb070 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:15.678807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:15.711617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:15.712598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:15.713062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:15.713504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:15.714355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:15.716788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:15.757757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 341/696] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:22:15 2022] POST /identity/v3/users => generated 254 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:22:15.807587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 356/697] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:16.155721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:16.280282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6e8c7f-6b6e-4b51-9d13-81be03b34c3b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:16.318258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:16.318945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:16.319223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:16.319487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:16.334262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:16.340758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:16.341916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:16.342793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 342/698] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:16.351721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:16.504208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:16.541982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3433ee-7235-411d-a9dd-0ebc45e9c3c8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:16.598343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:16.599000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:16.599299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:16.599569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:16.600256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:16.600560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:16.602080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:16.612573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 357/699] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:16.623680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:16.673525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:16.786619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87542e7-a8e6-4bcd-9d4b-f47020f11cbe None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:16.833932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:16.834589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:16.834932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:16.835283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:16.853934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:16.855504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e1911650-b96f-4bc8-bcd6-f87842992e74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:16.877124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:16.879473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:16.881692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:16.881692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:16.881692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:16.881692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 343/700] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:16.888964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.060907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6fe56f-5ec0-4b0b-a66d-41613d0ed680 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:17.091062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d9ec-9a23-4710-ab1a-4a1249a94ad6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.091391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d9ec-9a23-4710-ab1a-4a1249a94ad6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.091598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d9ec-9a23-4710-ab1a-4a1249a94ad6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.093242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 344/701] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:17.100157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.100443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.100862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.119505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:17.120306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.120579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.120845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.121503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:17.121801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:17.123210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:17.130873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:17.143526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:17.144743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:17.150733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 358/702] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:17.159649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.237428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 345/703] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:22:17.331997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299dedd0-8af8-4821-aba4-707cc68c7486 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:17.375341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:17.377037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.377548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.378012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.393448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:17.395556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-13f99bd1-8c0b-4302-ae58-d8a3935486e6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:17.410561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:17.411782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:17.412707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:17.415262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 359/704] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:22:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:17.423954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.424751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f71e882-021e-4302-b4be-ad0d1e9655d5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.425009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f71e882-021e-4302-b4be-ad0d1e9655d5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.425281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f71e882-021e-4302-b4be-ad0d1e9655d5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.426900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 360/705] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:17.433155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.433544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.433831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.442957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:17.473898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:17.475226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:17.561147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c976a0-d701-4fb6-971e-7446d3a738a4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:17.563450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:17.582470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 361/706] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:22:17.583518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.601447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:17.602574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.603124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.603588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.604528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:17.605034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:17.606786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:17.629006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 346/707] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:22:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:17.634473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.689866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203080e7-0523-4b02-9f2d-481dd03fc40f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:17.715311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:17.732496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:17.733238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.733711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.734052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.745996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:17.747540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:17.753386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1623b-a7b5-403f-92cf-d19a2ba7e4bc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:17.767727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:17.768794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:17.769673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:17.777361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 362/708] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Oct 4 08:22:17 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:17.783559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:17.821112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:17.822332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:17.822814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:17.823089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:17.875913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:17.912271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:17.913735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:17.975273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11267032-3b30-4dfd-8f15-5012aca4b453 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.007167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:18.018093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.019029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.019492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.019989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.020770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:18.021232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.022916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.028701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 363/709] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:17 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:18.029916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.053766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 347/710] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:18.053766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.181622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8cc9cc-7d87-49e6-a10a-9d3a209359c5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.189397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08cd919-62df-4cdd-ae86-596213100848 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.202725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:18.234496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.235367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.235831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.236022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.236256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.236519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.236944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.237166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:18.237463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.238252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] PATH_INFO: `/v3/services/network` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.239112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.239404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.241053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.248528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 348/711] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:18.249483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:18.255533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:18.256617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:18.257457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:18.263016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 364/712] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/services/network => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:18.272863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.334949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:18.456070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027d582-0a15-4438-8a66-fd778616eab8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.502230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc4e6e9-8a63-4de1-9d02-3b37faa46c11 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.512562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.512942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.513201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.513500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.525990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.527506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:18.542841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:18.544089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:18.545261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:18.545261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:18.545261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:18.545521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 349/713] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:18.550578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.553341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.554254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.554536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.554831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.555576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] RBAC: Adding query filter params (name=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:18.556056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.557970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.564570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 365/714] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:18.573113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.641518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:18.683315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff34090-67b2-4e32-a329-cc692c5fa86c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.694940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:18.735323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.736272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.736752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.737262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.738176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:18.738787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.740422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1185f0-238d-46a2-81b4-dc3c8c6dd757 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.741995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9217240-c43a-4153-8ede-aeb17259018b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.757312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 350/715] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:18.759716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ee99a-a9b8-4e0c-ac40-c8fa7c22b3c0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.760239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ee99a-a9b8-4e0c-ac40-c8fa7c22b3c0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.760733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ee99a-a9b8-4e0c-ac40-c8fa7c22b3c0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.762663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 351/716] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:18.763715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ee99a-a9b8-4e0c-ac40-c8fa7c22b3c0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:18.796065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:18.796678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.796956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.797223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.798139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] RBAC: Adding query filter params (type=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:18.798439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:18.800598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206603d8-b936-4866-92d5-faf1ec791138 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:18.808873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 366/717] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/services?type=network => generated 124 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:18.810188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:18.810472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:18.810830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:18.836329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:18.838759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:18.903762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ee99a-a9b8-4e0c-ac40-c8fa7c22b3c0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:18.982212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 367/718] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:19.001541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.002623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.003192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.003667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.023938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:19.025684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-05d8aba9-0149-4e5d-9546-e414040499e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:19.048784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:19.049748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:19.050750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 352/719] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:19.058571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.107148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:19.114441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03c9e94-45f3-4818-aab8-173759b52b32 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.114776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03c9e94-45f3-4818-aab8-173759b52b32 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.115006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03c9e94-45f3-4818-aab8-173759b52b32 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.116603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 353/720] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:19.131213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.131644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.132043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.154163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:19.175630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:19.176848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:19.206461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74544ad-8e72-43c3-9256-c0b84c80b9a3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:19.291923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 354/721] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 04 08:22:19.307874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.308750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.309247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.309667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.310528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:19.311013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:19.312644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:19.325939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.337164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 368/722] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:19.343750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.445095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d1acb-11a8-4125-8530-6c2b7976876a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:19.485074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.485766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.486041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.486313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.489416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:19.500472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:19.506110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:19.507292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:19.508405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:19.511712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 355/723] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:19.517852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.575273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:19.592859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a8ff85-b90c-43a0-bc7f-9d8f5e0125c2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:19.637988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a5ebd5-a7bf-46f3-b50b-af143548d710 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:19.675117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.676007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.676429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.676840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.677643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:19.678083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:19.679306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.679808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:19.679925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.680192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.681008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.682051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:19.682460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:19.684130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:19.691583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 356/724] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:19.696798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 369/725] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:19.698256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.772280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:19.808401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:19.836548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144489a-8920-4900-bc03-6a2a6e3298c9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:19.885960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:19.886661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:19.886942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:19.887213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:19.910846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:19.912330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:19.941058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:19.942046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:19.942952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:19.942952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:19.942952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:19.948402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 357/726] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:19.949401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.027313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ce35a-4667-46a2-aabc-3200279d7fe0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.075549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.076432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.076851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.077288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.105678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:20.116613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:20.117793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:20.119685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:20.125682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 370/727] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:20.134728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.167127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd0bcda-3165-40b1-94f0-501b52a53d29 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.179112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:20.191520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:20.251675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.252658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.254188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.254879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.257234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:20.257813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.259831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.298208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 358/728] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:20.307494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.325819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86298978-a841-4ae6-bba2-632d2b175d2f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.366140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.367148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.367612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.369359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.371048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:20.371782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.373480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.386932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 371/729] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:20.395696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.537597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:20.548613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4b751a-f41b-4ee3-93e2-7e5c9cc95e2b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.560913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdc8f89-9616-4b75-9095-058b5442a31e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.591048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.592098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.592591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.593083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.600841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.602693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.613550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.614207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.614461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.614766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:20.616868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:20.617769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:20.620075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:20.622983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 359/730] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:20.632632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.636986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.638587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:20.666163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:20.667190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:20.668117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:20.668117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:20.668117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:20.668117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 372/731] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:20.676557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.686018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:20.801089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0772b-739c-4502-b0d4-c5d15d22a88d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.845388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.846248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.846896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.847339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.849137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:20.849883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.853597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.856702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:20.881335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 373/732] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:20.888863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:20.892097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61a0037-2a29-409a-a613-71e3b026314b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:20.933021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:20.936018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.936018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.936018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.936299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:20.937019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:20.939402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11034ec7-f3ee-444a-9881-8d257e0792a0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:20.967341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 360/733] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:20.967341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c6045-9cb4-409b-a15b-0f63392ff401 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:20.967341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c6045-9cb4-409b-a15b-0f63392ff401 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:20.967341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c6045-9cb4-409b-a15b-0f63392ff401 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:20.967341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 361/734] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:20.968854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911c6045-9cb4-409b-a15b-0f63392ff401 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.033084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035dcc8f-f419-413d-a2cd-d14c0a605637 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.073546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.074597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.075889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.076509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.077265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.086637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.088668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.090099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911c6045-9cb4-409b-a15b-0f63392ff401 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-78bb3a44-ecb6-477b-9bab-9321c17e8801 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:21.103007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:21.104017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:21.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:21.108350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 374/735] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:21.109944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.110465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.111016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.132019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:21.133592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:21.136035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.136646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.136922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.137188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.217664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.219218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f4226c-0318-498e-8c1e-e5827dff1e1d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.222376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.231008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 375/736] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:21.232754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.363841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57688ee0-0752-4660-861b-eba70c0a631f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.373506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.390177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 362/737] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:21.391909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6523e-b7e9-40e4-bce4-da4afc8c4ca6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.392354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6523e-b7e9-40e4-bce4-da4afc8c4ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.392800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d6523e-b7e9-40e4-bce4-da4afc8c4ca6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.398560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 363/738] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:21 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:21.400200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6523e-b7e9-40e4-bce4-da4afc8c4ca6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.416900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.417990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.418587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.419165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.420513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:21.421147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.423846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac354bc-d51b-4799-894d-e8ada0a98d06 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.461497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 376/739] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:21.464287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.464848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.465384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.493903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:21.495323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:21.532024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d6523e-b7e9-40e4-bce4-da4afc8c4ca6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.550630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.574025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.574936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.575294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.575621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.589286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 377/740] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 04 08:22:21.590206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.591948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.593859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:21.613607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:21.614862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:21.616078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:21.616078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:21.616078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:21.619210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 364/741] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:21.620485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.679334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.711294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26920550-ffb1-440a-8298-cae102eac7e1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.753905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.756611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.756611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.756611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.777195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde5c78f-fd10-4ecc-b549-59ee0fa3efd8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:21.800814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:21.819725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:21.821281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:21.821750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:21.822477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:21.823446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.825158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.831310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:21.833680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:21.850480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 365/742] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:21 2022] POST /identity/v3/services => generated 251 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:21.853075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:21.971672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 378/743] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 04 08:22:21.972514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.001305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.025593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74aac54-bcf4-4caf-8868-1c7178a8a39a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.066251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.079338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.080613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.081107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.081407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.082177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:22.082512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.084244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.098023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 366/744] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:22.098744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.104185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f5df70-f600-4f6a-b5a8-d56f6327238b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.148905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.150594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.151241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.151847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.152865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.154516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.178638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 379/745] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:21 2022] POST /identity/v3/services => generated 240 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:22.179743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.242034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37411b1e-6d41-45d2-a369-78543381eaeb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.305158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.307012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.307736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.308383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.313540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61123-2462-4868-bb29-a536a7374b7f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.328024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.330939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.349691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.356046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9fc09fd4-cbf8-4cdc-8853-bc3a28a49db4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:22.356433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:22.357369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:22.358340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:22.358340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:22.358340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:22.358513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.358513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.358678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 367/746] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:22.358747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.359977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69423afa-5d95-4ce0-80a8-0882c88d4a31 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.359977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69423afa-5d95-4ce0-80a8-0882c88d4a31 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.359977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69423afa-5d95-4ce0-80a8-0882c88d4a31 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.366067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 368/747] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:22.367395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69423afa-5d95-4ce0-80a8-0882c88d4a31 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.372096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.375221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.376917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-496898f8-85a0-44a8-b263-08c9cbb88373 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:22.396157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:22.398143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:22.400415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 380/748] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:22.407339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f71df-badd-4961-81d6-8de288b47919 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.407745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f71df-badd-4961-81d6-8de288b47919 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.408125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930f71df-badd-4961-81d6-8de288b47919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.444055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-930f71df-badd-4961-81d6-8de288b47919 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:22.446036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-930f71df-badd-4961-81d6-8de288b47919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:22.530222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69423afa-5d95-4ce0-80a8-0882c88d4a31 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.535145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 381/749] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:22.536007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f71df-badd-4961-81d6-8de288b47919 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.578577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.578577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.578577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.578577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.579090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:22.579090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.579090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.604077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 369/750] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:22.615586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.690108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930f71df-badd-4961-81d6-8de288b47919 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.722907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.745326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.746475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.747022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.747353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.748998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:22.749313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:22.749593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.751178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.781460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 382/751] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:22.788148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.834673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d05ccca-8c79-42a9-b84d-6a37d8c3172f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.897796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.898609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.899011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.899785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.910310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.912721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:22.927407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:22.928532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:22.929646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 370/752] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:22.931926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:22.933775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02a1a9-9f0e-45c4-9e99-502bb19c9115 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:22.973436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:22.974245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:22.974738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:22.975142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:22.976099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=5cc208be4d444563929a03634cfddee7) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:22.976566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:22.978516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:22.985183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 383/753] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/role_assignments?user.id=5cc208be4d444563929a03634cfddee7&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:22.986220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:23.099040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:23.133804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82799d59-5032-425e-af9f-1b7e7bf6f95d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.190846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:23.206260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.207469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.207968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.208466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.222244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:23.226363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d61107-ac52-4ced-83e5-8d4082337ffc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:23.229280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:23.230234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:23.231420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:23.233813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 371/754] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:23.239415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:23.335404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.336156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.336438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.336717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.337488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:23.337796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:23.339507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:23.354221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0859557d-3ec6-475e-8a3e-abde979f5703 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.374763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 384/755] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:23.380821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:23.392079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.392700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.392981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.393256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.393926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:23.394229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:23.396322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:23.411115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 372/756] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:23.412979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:23.470970 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:23.531170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfd2922-21a0-48ee-824f-088ebd7b1b1d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.582617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.583277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.583544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.583816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.607268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:23.609448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:23.623123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b45f1-47a7-4b9c-b7db-d8ae4794c889 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.638921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:23.640331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:23.641240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:23.642115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:23.642115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:23.642115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:23.643259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 373/757] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:23.650292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:23.700124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.700792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.701073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.701342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.702073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=908ed003368445ae8db242549c212981) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:23.702377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:23.703860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:23.711520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 385/758] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:23 2022] GET /identity/v3/role_assignments?user.id=908ed003368445ae8db242549c212981&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:23.861271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:23.866945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf30a4b4-11be-4ca0-b358-d7f6fc7f9448 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:23.925245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:23.925872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:23.926145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:23.926409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:23.927091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:23.927388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:23.928790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:23.950384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 374/759] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:23.956177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:24.121606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c959b3-a2c4-47fa-af35-74f087e603a7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:24.169685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:24.170332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.170637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.170904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.183631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:24.185154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:24.200459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:24.201455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:24.202373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:24.204800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 386/760] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:22:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:24.211195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:24.376599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408fc019-6c5a-4374-a89d-624cdba2bab9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:24.434521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:24.435559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.436036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.436518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.437448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:24.437952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:24.439619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:24.470302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 375/761] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:22:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:24.476003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:24.520551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:24.610300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e40d39-53c8-481e-aa59-38ac5fe857be None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:24.660429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:24.661003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.661278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.661548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.674268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:24.675770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0a9b7805-f658-4664-9377-7aad5693d125 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:24.690052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:24.691077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:24.691969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:24.691969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:24.691969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:24.691969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 387/762] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:24.699628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:24.703854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40590-5cba-4c8e-8955-01fb864b9b61 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.704126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40590-5cba-4c8e-8955-01fb864b9b61 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.704388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40590-5cba-4c8e-8955-01fb864b9b61 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.705703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 388/763] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:24.719057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.719324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.719597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.724353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:24.742130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:24.743664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:24.851136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31310c7-9baa-48aa-ad7a-8096f231e815 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:24.887897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:24.888509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:24.888785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:24.889051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:24.889692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:24.889994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:24.891375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:24.903363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 376/764] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:24.908836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:24.913138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 389/765] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:24.988600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.013087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d362a7-4781-4ed6-a962-1bed072d4e4e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.049105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.049763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.050038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.050304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.059357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.060805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b37feed7-9c55-48c9-a927-2f44080da905 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:25.073731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:25.074736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:25.075615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 377/766] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:25.081308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.106330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.240507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68346e2f-bd35-4d86-bd13-3368b94c8571 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.291623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accd6fb0-e177-41d5-ace8-f15e938dbe23 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.291944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accd6fb0-e177-41d5-ace8-f15e938dbe23 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.292167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accd6fb0-e177-41d5-ace8-f15e938dbe23 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.293764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 378/767] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:25.296621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.297335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.297728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.298066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.298997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:25.299381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.301393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:25.308605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.308902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.309149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.318755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 390/768] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:25.319826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.330868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:25.332498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:25.406343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.427622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 379/769] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:25.428659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.470712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246fcb03-51cf-45a2-9465-dd47aa06b42e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.545174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.546071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.546494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.546819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.547417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.548801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:25.561047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:25.562341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:25.563273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 391/770] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Oct 4 08:22:25 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:25.569137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.574911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.639107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a62a55-5f93-49c2-a7d1-61ed8ef0205d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.679784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.681430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.682259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.683109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.686413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=02d91fccc1c8400392fbbc34416f552b, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.688497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d6553-e8db-4f03-aff9-741fb56cf807 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:25.705013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 380/771] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:25 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:25.707750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c52c4-1e73-4a22-9bc0-4fa3d9341377 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.708199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c52c4-1e73-4a22-9bc0-4fa3d9341377 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.708623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4c52c4-1e73-4a22-9bc0-4fa3d9341377 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.709975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8a032a-d7a3-4cb7-a23c-04be5a736d20 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.710818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 381/772] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:25.726518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.726874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.728019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.762710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:25.763953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:25.775246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.776003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.776340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.776611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.777362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:25.777663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.779586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:25.783018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.788222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 392/773] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:25 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:25.789108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.848792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 382/774] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:25.849640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:25.941518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036ae8c-c6ce-49df-b379-f8d1ba9dce09 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:25.954668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:25.987272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:25.987963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:25.988246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:25.988514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] PATH_INFO: `/v3/services/network` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:25.989103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:25.990491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:26.000770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:26.001796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:26.002663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 393/775] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:25 2022] GET /identity/v3/services/network => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:26.018532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.018750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c37dac-fe04-41c0-b3d5-44df7c553301 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.064404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.065023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.065293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.065561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.066208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.066501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.068960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.081146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 383/776] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:25 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.087654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.133385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.214378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf19808-ccde-41f8-8834-e93ce283313f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.253316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd9c599-04a1-4ecf-b17c-2e3cd6d7bded None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.267806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.268720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.269208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.269639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.270641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] RBAC: Adding query filter params (name=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.271117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.273482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.276129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.282186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 394/777] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/services?name=network => generated 124 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.283854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.300061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.300679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.300955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.301323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.302033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] RBAC: Adding query filter params (interface=public, service_id=b8f188d38e3c4ce8b1581e2add92aa3c, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.302337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.303762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.313638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 384/778] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/endpoints?service_id=b8f188d38e3c4ce8b1581e2add92aa3c&interface=public®ion_id=RegionOne => generated 194 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.314675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.429025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c8d845-baa1-4981-add3-94359e49ce5f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.469437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.479169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.479802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.480074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.480339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.481420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649235fb-d925-4eb0-bef3-e8350369f843 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.492139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:26.498225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:26.499231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:26.500070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:26.504382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 395/779] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:26.505294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.529236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.529855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.530136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.530411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.531089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.531942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.533704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.540420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 385/780] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/services?type=network => generated 350 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.542051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.615922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.631431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d562b1af-b73d-4982-aebd-6291ec10bc99 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.670900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.671508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.671820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.672090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.672690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.674068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.677353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2cb8b-92e5-4957-815a-6cc044f1b0f4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.682793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 396/781] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.682793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.737827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.738442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.738750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.739022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.739698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.739997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.741582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37716792-8826-4094-bb9f-6abe6617a258 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.758304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.758694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 386/782] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.761076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e214570-a60d-4205-b535-7549d682e185 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.761512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e214570-a60d-4205-b535-7549d682e185 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.761786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e214570-a60d-4205-b535-7549d682e185 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.764017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 387/783] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:26.764017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e214570-a60d-4205-b535-7549d682e185 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:26.834075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743dd12b-7699-41f2-b164-26700af0228e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.887638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.888412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.889110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.889498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.890409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] RBAC: Adding query filter params (interface=public, service_id=a13c42d456314353b0046e1868c63911, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:26.891471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.893148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7b7af0-2cdc-4945-afc1-e6294b5957b9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:26.904718 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 397/784] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/endpoints?service_id=a13c42d456314353b0046e1868c63911&interface=public®ion_id=RegionOne => generated 194 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:26.906794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.909492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.909492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.920849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e214570-a60d-4205-b535-7549d682e185 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:26.927887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:26.929088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:26.935073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:26.963394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:26.964608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:26.965215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:26.965966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:26.982379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:26.984892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.002076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 398/785] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:27.003046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:27.005128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:27.006252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:27.007224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:27.007224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:27.007224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:27.009643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 388/786] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:27.023441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.056938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:27.113297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc2d139-2114-44c1-991b-03b73a8ffaeb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.159326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.160052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.160429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.160818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.161538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:27.163188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.169287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 399/787] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/services => generated 772 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:27.219514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c87c4a-045e-48bd-97ac-2c8143af6508 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.263060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.263695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.263972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.264240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.264959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:27.265261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:27.266734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.291396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 389/788] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:27.297107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.309104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.361697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:27.445949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793afa-f189-4fa5-a929-1665743f2419 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.459156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:27.470747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafab64-9972-4a98-a26c-cf71a3089f8b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.487417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.488064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.488338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.488605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.498472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:27.499968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:27.512556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:27.513574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:27.514623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 400/789] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:27.518617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.519449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.519849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.520137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.523523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.544732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:27.550823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:27.552978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:27.554695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 390/790] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:27.565211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.732020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:27.749981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0f9f9-af99-436f-aebc-efc421377c64 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.779826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8bdcc7-7ff1-495d-9718-1ae56e6361db None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:27.808504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.809017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.810189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.810468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.811236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:27.811541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:27.813012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.850434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:27.851075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:27.866591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:27.866591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:27.866591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:27.866591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:27.866591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:27.867468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 401/791] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:27.867468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.879557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 391/792] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:27.886374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:27.917993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.069252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7634485-35ae-4581-b313-2dc452ee8051 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.094234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30625e7-520b-4580-92c9-6b3d30f3f2d1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.147146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.147792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.148071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.148338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.160859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.162334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:28.179097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:28.180112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:28.181564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:28.181564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:28.181564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:28.181564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 402/793] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:28.187597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.194157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.194839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.195118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.195383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.216620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.219896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.226137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:28.247582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:28.248636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:28.249505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:28.249505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:28.249505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:28.252201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 392/794] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 365 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:28.256322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.348578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.380807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da987b-8f9c-4e02-b5ac-7354ba8b629d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.383867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bef5988-32bb-49e6-9a63-20380a047ddc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.420800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.421836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.421949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.422443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.422584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.422774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.423049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.423453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.423701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:28.424003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.424726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:28.425224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.425395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.426997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8de6ee8-204f-4994-b69d-552a96f82914 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.440490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 393/795] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:28.442815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 403/796] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:28.444314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb68669d-3f5e-4cdb-b2b1-3118b56dd92f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.444815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb68669d-3f5e-4cdb-b2b1-3118b56dd92f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.445309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb68669d-3f5e-4cdb-b2b1-3118b56dd92f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.449775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.451552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 404/797] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:28 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:28.453728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb68669d-3f5e-4cdb-b2b1-3118b56dd92f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.534709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.569902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb68669d-3f5e-4cdb-b2b1-3118b56dd92f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.588379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3678-ee5a-48d0-9a4a-ea45aef9c965 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.629839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.630934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.631492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.631966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.641770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.643610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-64971c69-59ed-46e5-945f-ba00003239af None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:28.657109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:28.658104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:28.659090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:28.661955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 394/798] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:28.663554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.664055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.664525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.685328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:28.686933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:28.731050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.731703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.732034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.732358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.740054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.741418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.745560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:28.759656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:28.760656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:28.762035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:28.764692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 405/799] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:28.765686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.786842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 395/800] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:28.787934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:28.893135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb513db-70fd-4860-a389-84490da40e29 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:28.911994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:28.942788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:28.944901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:28.945782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:28.948001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:28.948901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:28.949570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:28.953385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:28.998141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 406/801] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:29.002508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.048218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c5122-3675-4b85-8fd7-3dfa3f39fb9c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.121421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.153643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896be401-0bde-495a-9728-5c44868ca4ba None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.158875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.159512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.159790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.160060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.160771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:29.161079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.162521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.180714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 396/802] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:29.182686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.202947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.203645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.203916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.204196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.216857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.218306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:29.232809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:29.233827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:29.234758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:29.234758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:29.234758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:29.234758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 407/803] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:29.235452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.308566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484c19-2d4d-4e29-b163-e855bd74bc9a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.316060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.351616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.352290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.352632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.353661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.360286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df2f5a5-5148-474c-9df3-8ec25f4bd389 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.372940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:29.373567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b09c8e95-4eae-4bd1-aef6-593e48476b7d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:29.383135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:29.384675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:29.385510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:29.386919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 397/804] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:29.387679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5ddef-8e37-44d8-b60f-c781e0799d40 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.388110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5ddef-8e37-44d8-b60f-c781e0799d40 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.388519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5ddef-8e37-44d8-b60f-c781e0799d40 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.396516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 398/805] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:29 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:29.397756 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d5ddef-8e37-44d8-b60f-c781e0799d40 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.405206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.405825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.406098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.406365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/5cc208be4d444563929a03634cfddee7/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.408259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=5cc208be4d444563929a03634cfddee7, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.409680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.425817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 408/806] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:29 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/5cc208be4d444563929a03634cfddee7/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:29.427069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.552741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c170a8-ab88-4c58-8f89-395d627808ad None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.597347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.598448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.598993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.599468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.600387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:29.600911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.602879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a24d28-919b-4639-9476-d6299a9f8975 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.613577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 409/807] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:29.615463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.615975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.616463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.638999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:29.640535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:29.673168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.687814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d5ddef-8e37-44d8-b60f-c781e0799d40 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.726774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 410/808] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:29.728290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.770503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.771438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.771865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.772286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.773300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:29.773764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.779339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.805240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 399/809] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 408 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:29.810382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.851186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.867461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd7d3ac-4e22-4d55-b037-bd693aad0404 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.907967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.908653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.908935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.909208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.920979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:29.922449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:29.936514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d611d9-7f54-4b95-b017-873564e2c0eb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3e91e8fa-166f-4e21-804b-effe46022446 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:29.941422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:29.942348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:29.943287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:29.943287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:29.943287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:29.947780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 411/810] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:29.949095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a2f58f-3a15-4e57-9d85-9a690cbb41e5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.949375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a2f58f-3a15-4e57-9d85-9a690cbb41e5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.949644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a2f58f-3a15-4e57-9d85-9a690cbb41e5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:29.951212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 412/811] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:29.951917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a2f58f-3a15-4e57-9d85-9a690cbb41e5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:29.968618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:29.996678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:29.997312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:29.997578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:29.997842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.007354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.009664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-67071b35-c914-43b6-ac46-052d8f337b28 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:30.021784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:30.023697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:30.025188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 400/812] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:30.025878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.025878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.026034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.051401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:30.052849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:30.070504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a2f58f-3a15-4e57-9d85-9a690cbb41e5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.116888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.117774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.118822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.119245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.120549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:30.121274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.123117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.139671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 413/813] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:30.141379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.146579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 401/814] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:30.151847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.212785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.234528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.281138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e0531-9610-4699-ad52-015a3f02cb9c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.326817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.327677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.328131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.328799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.329676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:30.330766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.332698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.352326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 414/815] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:30.352326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.366298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adff3c2-3e8c-4b72-8e70-40531c2aeed4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.409360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.410344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.410868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.411882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.421901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.423757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:30.438565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:30.439632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:22:30.440569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:30.443820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 402/816] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:30.445483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.498949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a31782c-d468-4013-bd01-96b3fe81b091 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.536803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.537415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.537687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.537950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.538528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.539083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.539963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.542029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:30.549769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:30.550869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:30.551803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 415/817] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:30.552628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.573487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce70651-a35b-4bc5-8ab5-feee2897646f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.612536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.613188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.613460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.613722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/908ed003368445ae8db242549c212981/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.630724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=908ed003368445ae8db242549c212981, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.632293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.646589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 403/818] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:30 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/908ed003368445ae8db242549c212981/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:30.647401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.680484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49040aeb-57e6-44ae-b871-b70b3589b500 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.748904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.749967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.750466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.751062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.752133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:30.752657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.755117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.786797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48af3468-15f2-4a33-bda5-578a669803b5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.787044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 416/819] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:30.790711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.818142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.833067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.833731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.833974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.834239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] PATH_INFO: `/v3/services/network` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.834852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.836250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:30.849283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:30.850323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:30.851626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 404/820] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/services/network => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:30.852396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:30.922992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:30.935526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1206ef49-51cd-4f94-9419-5a8531111f56 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.985010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:30.985622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:30.985896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:30.986163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:30.986963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:30.987263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:30.988626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:30.990700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9ddc8f-1963-429c-a990-3e88f7c7e12e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:30.994688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 417/821] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:30.995522 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.034574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.036130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.036585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.037047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.117466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5a3043-6287-44e7-a432-a04287de85af None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.123910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.143372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.143372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.153676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.176909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.178013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.178596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.179130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.179870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] RBAC: Adding query filter params (name=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:31.180271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.182396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.187892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 418/822] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/services?name=network => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:31.188815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.281859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 405/823] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:30 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:22:31.282859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.301035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4929ff3-15e3-4e92-9752-286b0cd734da None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.343711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.344390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.344673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.344948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.345585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:31.345888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.347308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.355549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 419/824] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Oct 4 08:22:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:31.356389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.405712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.421424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aab0c3-8469-44d9-984d-0d811f71e932 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.425819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.483456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.485060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.486082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.486913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.488101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] RBAC: Adding query filter params (type=network) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:31.488922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.491182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.491784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593ceffe-896a-4928-a2ce-b04b6d452ead None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.501404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 406/825] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:31 2022] GET /identity/v3/services?type=network => generated 350 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:31.503681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.541348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.542475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.542918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.543335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.555417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.558172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:31.574702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:31.575885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:31.576958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:31.576958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:31.576958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:31.578381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 420/826] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:31.579403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.628784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19cfc77-da25-4aea-a12e-2d79b1dcdd93 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.665618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.666254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.666533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.666834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.667532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.669102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.695841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 407/827] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:31 2022] POST /identity/v3/endpoints => generated 346 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:31.697383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:31.703284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.707549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37417c7e-c2ef-4389-b43e-15cc6619a73e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.716395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:31.800946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.801428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.801704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.801968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.802663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.804042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f591d75-c3be-47e8-b119-ef61271f56d9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.829302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b0240f-c853-49ae-a30b-978dfd10c6a6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:31.838500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 421/828] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:31 2022] POST /identity/v3/endpoints => generated 337 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:31.839853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12cc11-1598-4910-ac65-7ecddcd7a55a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.841537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12cc11-1598-4910-ac65-7ecddcd7a55a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.841674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d12cc11-1598-4910-ac65-7ecddcd7a55a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.851241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 422/829] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:31 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:31.856700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.857012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.857250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.887187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:31.888290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:31.888790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:31.889305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:31.890232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:31.892598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:31.894916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:31.910815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:31.912897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:31.933090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 408/830] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:31.938481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.035746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 423/831] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:32.068991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:32.116372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fa6b18-906c-4ffe-8f6c-3b0df9feb654 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.163351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.163989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.164261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.164524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.173522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:32.175266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:32.187672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:32.188592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:32.189499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:32.191556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 409/832] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:32.199452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.309307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:32.333710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998102e0-dfdd-4081-a882-28da3d77f665 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.340675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.374224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.375192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.375637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.375908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.376626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:32.377177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:32.378435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:32.392506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 424/833] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:32.398311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.466124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee86ff-7784-4a22-b694-4282413b35c9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.498901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:32.511724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.512567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.512984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.513389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.534955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:32.542581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:32.543825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:22:32.544715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:32.547762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 410/834] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:32.554008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.561394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5de58-6c0f-4c4c-aec2-3420c381e6cb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.605015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.606004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.606487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.607018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.607933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:32.608443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:32.610066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:32.616987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 425/835] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:32.682517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8915389-adb9-4264-8275-f9257c1340ea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.722671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.723570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.724004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.724431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.725280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:32.725747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:32.727332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:32.739356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 411/836] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:32.746249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:32.763803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:32.871305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473952a2-ffc9-420c-9dc0-ab7976e6c6a2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:32.911672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:32.912328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:32.912608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:32.912878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:32.925005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:32.926467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-da9f067f-eb23-490a-babc-03853a843312 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:32.941842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:32.942908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:32.944233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:32.944233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:32.944233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:32.944233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 426/837] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:32.948699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.012267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.067755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d8001e-064c-4aea-9d92-6b105b3484ba None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.104084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.104692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.104966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.105236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.105875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:33.106174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:33.107568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:33.120593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 412/838] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:33.123360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ce4c-8354-4ecf-89fb-9b4ad4dcc4a2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.123647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ce4c-8354-4ecf-89fb-9b4ad4dcc4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.123924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ce4c-8354-4ecf-89fb-9b4ad4dcc4a2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.125557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 427/839] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:33.125865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.130818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.131098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.131371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.153315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:33.154591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:33.219332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.237012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 428/840] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:33 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:33.253702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb5677b-3c97-4018-a0e2-61b6c52f66e1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.264227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.292567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.293470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.293897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.294315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.302349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:33.303987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:33.316335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:33.317531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 04 08:22:33.318833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:33.321069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 413/841] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:33.327161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.433607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcae166a-d8b4-4c64-bb31-a0a54e37776b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.469624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.470155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.470493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.470793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.471568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:33.471871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:33.473400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:33.493319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 429/842] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:33.500641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.511535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.578932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.628859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c997c-5acd-433a-8fbd-966f16ce9154 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.651808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f8215a-b14e-49c1-8f21-f340c0eb4f16 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.664542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19886781-2e93-424e-8259-131cf05350bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.666033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19886781-2e93-424e-8259-131cf05350bf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.666491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19886781-2e93-424e-8259-131cf05350bf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.667255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19886781-2e93-424e-8259-131cf05350bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.698858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.699837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.700211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.700489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.707721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.718769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:33.723667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:33.724596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:33.725487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 430/843] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:33.731953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.758434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19886781-2e93-424e-8259-131cf05350bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:33.762202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19886781-2e93-424e-8259-131cf05350bf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:33.876391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485fe097-a4bb-4b99-8875-7de7f94c18f5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:33.889534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:33.922807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 414/844] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:22:33.928518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19886781-2e93-424e-8259-131cf05350bf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:33.935531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:33.936450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:33.936876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:33.937303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:33.938202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:33.939378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:33.941411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:33.959182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 431/845] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:33.961183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.018444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.067402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19886781-2e93-424e-8259-131cf05350bf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.086277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940052c-7d43-45a2-bebd-fcc2e048c188 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.123789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.125053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.125772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.126169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.133255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae641294-623b-48f2-a970-32e913119350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.134745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae641294-623b-48f2-a970-32e913119350 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.135227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae641294-623b-48f2-a970-32e913119350 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.135787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae641294-623b-48f2-a970-32e913119350 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.148477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.150618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae641294-623b-48f2-a970-32e913119350 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.152351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae641294-623b-48f2-a970-32e913119350 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.156381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ae641294-623b-48f2-a970-32e913119350 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:34.171003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:34.172063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:34.172988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:34.172988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:34.172988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:34.173161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:34.174152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:34.175071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:34.175071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:34.175071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:34.182577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 415/846] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:34.182577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.183377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 432/847] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:34.184871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae641294-623b-48f2-a970-32e913119350 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.216513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.310716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044e3181-7e5c-4e4b-999a-01f7279d9516 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.318939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae641294-623b-48f2-a970-32e913119350 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.332670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.349586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.350212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.350489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.350783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.351558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:34.351865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.353541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c970f08f-f6a6-4085-acf3-9e4eafa52350 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.359905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.360557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.360777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.361043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.361687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:34.361978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.363377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cec8aaa-2b11-471e-8fd2-380e333daaea None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.367097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 416/848] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:34.368900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8fd07-b19d-46c3-ab58-b8fa564657d5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.369608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8fd07-b19d-46c3-ab58-b8fa564657d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.370091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8fd07-b19d-46c3-ab58-b8fa564657d5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.371500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 417/849] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:34.373014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8fd07-b19d-46c3-ab58-b8fa564657d5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.380609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 433/850] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:34.381885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.382301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.382606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.403425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:34.404816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:34.483507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 434/851] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:34.484490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.501447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.531359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8fd07-b19d-46c3-ab58-b8fa564657d5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.591735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.592381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.592718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.593058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.600468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.601994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.609454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f341946d-42fb-40dc-a754-3cb011d9fba9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:34.614314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:34.615626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:34.617108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:34.619546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 418/852] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:34.622404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.657938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.659051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.659535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.660087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.669422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.671206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.676398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:34.687345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:34.688307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:34.689321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:34.691587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 435/853] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:34.706573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.758389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a68f4d-6238-4366-9835-4fefe0fe06f8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.785017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.804846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.805702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.806680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.807057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.807885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:34.808267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.809809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.829394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 419/854] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:34.830193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0288464d-4e20-462a-9951-f94a7c6b975c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.830886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.870864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.871531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.871833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.872036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:34.872823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:34.873131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:34.874581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:34.897309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 436/855] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:34.897907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:34.956788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:34.960194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3baded6-0c33-48b9-9450-ee218ec00618 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:34.998128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:34.998767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:34.999043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:34.999313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.000032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=5cc208be4d444563929a03634cfddee7) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:35.000333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.002099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.010025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 420/856] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/role_assignments?user.id=5cc208be4d444563929a03634cfddee7&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:35.010025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.024455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152b3454-14cb-4f07-b347-3bfe7bffc72b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.070746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:35.084319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.085106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.085488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.085866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.086623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.088398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:35.101137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:35.102294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:35.104130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 437/857] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:35.108953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.187991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db98eff3-b140-45fe-8e39-4b44cf5cc4a9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.237663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.238322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.238669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.238942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.264644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ab43-8765-4dad-89d0-79ede2aaa82f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.277685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:35.303751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.304358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.304641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.304911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.305552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:35.306471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.308130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3d8f87-87e8-4bef-873c-d11ae6043347 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.317424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 438/858] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:35.317424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da4a5a-774a-4e3b-89e2-3aeb2b1c4d07 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.317424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da4a5a-774a-4e3b-89e2-3aeb2b1c4d07 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.317424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da4a5a-774a-4e3b-89e2-3aeb2b1c4d07 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.317424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 439/859] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:35.319806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da4a5a-774a-4e3b-89e2-3aeb2b1c4d07 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.329334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.330833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cf5bec-b78b-4f20-ba86-6ea13ce624ff None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.384025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:35.444442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da4a5a-774a-4e3b-89e2-3aeb2b1c4d07 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.455599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 421/860] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:22:35.457402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.457935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.458442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.481955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:35.483584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:35.484210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.484814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.485086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.485350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.485968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:35.486255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.487671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.492199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 440/861] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:35.492979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.577650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:35.598822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 422/862] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:35 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:35.638667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f371aec-ad8b-45a1-9a40-e2219f494059 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.664744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:35.688747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.689814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.690333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.690860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.703717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.705495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:35.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:35.723148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:35.724021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:35.724021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:35.724021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:35.726156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 441/863] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:35.732672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.845810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f0b402-48de-4774-95d9-f5dfadcca16e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:35.885894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:35.886931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:35.887424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:35.887905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:35.888750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:35.889254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:35.890866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:35.900348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.906396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 423/864] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:35.913842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:35.992127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:36.041494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3b291-534c-494f-b6cd-2c481867116d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.062411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80da400-db15-4bf7-a9bf-cdfa78585c87 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.085563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.086596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.087137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.088475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.089478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.091543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:36.103727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:36.104729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:36.105593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:36.111178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 442/865] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:36.113328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.113614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:36.114499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.114963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.115375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.125645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.127375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:36.143558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:36.144715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:36.145813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:36.147962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 424/866] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:36.154158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:36.177999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:36.266680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894b579-bf8d-4d6e-8e22-f5192f321f55 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.285761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622087d7-f0d8-4552-9634-80bf851e29c7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.302390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:36.313691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.314322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.314624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.314898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.315541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:36.315845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.317386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.325140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 443/867] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:36 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:36.333994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:36.334179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.334179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.334179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.334179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.334534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:36.334534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.335455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.349177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 425/868] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:36.359673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:36.458714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:36.465135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5151d8-da73-493d-a04a-b470f84a21a1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.485635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4cac8-1b41-4c77-a597-afdf6f4f17d2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:36.512038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.512940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.513385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.513723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.514419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:36.514818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.516807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702947f3-3775-4b6a-9b4f-e29fbed62655 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.528892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 444/869] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:36 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:36.532856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:36.533477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:36.533749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:36.534012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:36.534777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=908ed003368445ae8db242549c212981) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:36.535078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:36.536558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdffafd-96c2-46cf-a479-dbe90128ed9b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:36.544366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 426/870] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:36 2022] GET /identity/v3/role_assignments?user.id=908ed003368445ae8db242549c212981&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:37.375704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f8826-4fc7-40e6-86f1-2350b4710afc None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:37.375903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f8826-4fc7-40e6-86f1-2350b4710afc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:37.376178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f8826-4fc7-40e6-86f1-2350b4710afc None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:37.377767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 445/871] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:37.383436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea9ab3-abdc-4754-b2d3-926e3e987a43 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:37.384080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea9ab3-abdc-4754-b2d3-926e3e987a43 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:37.384080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ea9ab3-abdc-4754-b2d3-926e3e987a43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:37.403905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-80ea9ab3-abdc-4754-b2d3-926e3e987a43 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:37.405164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-80ea9ab3-abdc-4754-b2d3-926e3e987a43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:37.438011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:37.478986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 427/872] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:37.752070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89f8826-4fc7-40e6-86f1-2350b4710afc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:37.862077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951508f9-b0d3-499e-8da4-ac1e8ef72c62 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:37.862386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951508f9-b0d3-499e-8da4-ac1e8ef72c62 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:37.862687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951508f9-b0d3-499e-8da4-ac1e8ef72c62 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:37.864126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 428/873] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:37.868135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89f8826-4fc7-40e6-86f1-2350b4710afc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:37.869833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:37.870111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:37.870371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:37.894957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:37.896248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:37.926525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:37.927801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:37.928692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:37.929414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:37.930190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:37.933429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:37.945535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:37.947205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:37.948073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:37.950077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 446/874] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:37.953964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:37.988213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 429/875] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:37.989561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:38.068292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d4102-cf4a-4c13-a1b4-9d829ffb2e7e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:38.105984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:38.107047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:38.107553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:38.108044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:38.108889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:38.109389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:38.111015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:38.116293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 447/876] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:38.122645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:38.227271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:38.234396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85207dd0-f9e8-410d-9a3a-3ec227dfb6b0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:38.278635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:38.279245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:38.279519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:38.279785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:38.280411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:38.280708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:38.282132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a185a3d-9d76-419f-ba7f-e32bc8dac68c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:38.287208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 430/877] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:38 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:38.298612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:38.422973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:38.433589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5409aa-fe40-4bd4-9155-87f4a2535e20 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:38.473665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:38.474298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:38.474600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:38.474879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:38.475579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:38.476985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d850d2e2-4886-4abd-a700-9999a3ddac22 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:38.502764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 448/878] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:38 2022] POST /identity/v3/services => generated 251 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:38.878753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cf4e00-65ac-4ec0-9892-c82af964f6db None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:38.879271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cf4e00-65ac-4ec0-9892-c82af964f6db None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:38.879271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cf4e00-65ac-4ec0-9892-c82af964f6db None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:38.880901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 431/879] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:38.887457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbff9a-b97e-44dd-9c21-0dc64ed0bd6d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:38.887970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbff9a-b97e-44dd-9c21-0dc64ed0bd6d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:38.887970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cbff9a-b97e-44dd-9c21-0dc64ed0bd6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:38.912095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c0cbff9a-b97e-44dd-9c21-0dc64ed0bd6d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:38.913938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c0cbff9a-b97e-44dd-9c21-0dc64ed0bd6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:39.004787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 449/880] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:38 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:39.084424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c18bd5-434c-4bc7-bc10-a6504fa98be6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.084763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c18bd5-434c-4bc7-bc10-a6504fa98be6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.084974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c18bd5-434c-4bc7-bc10-a6504fa98be6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:39.086624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 432/881] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:39.092785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.093073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.093337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:39.114010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:39.115271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:39.197555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 450/882] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:39 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:39.293341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c18bd5-434c-4bc7-bc10-a6504fa98be6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:39.404078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c18bd5-434c-4bc7-bc10-a6504fa98be6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:39.440861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:39.441846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.442358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.442843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:39.443707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:39.445241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:39.461716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 433/883] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:39 2022] POST /identity/v3/services => generated 269 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:39.471502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:39.484382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:39.605015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e6c1f2-75e8-4895-a1f3-e14eef542326 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:39.650292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:39.650966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.651252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.651588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:39.662371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:39.667373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:39.669076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:39.670910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:39.674180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 451/884] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:22:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:39.676227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:39.706952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:39.793152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d23b9-1bb9-4609-93b4-1fc4a52abdc6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:39.828851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:39.829458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.829730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.829997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:39.830693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:39.830990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:39.832342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:39.846762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 434/885] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:22:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:39.848319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:39.917187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:39.957631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a83cfc2-eabd-4501-93a6-3a7e1c850760 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:39.992282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:39.993228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:39.994503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:39.994970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.005418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:40.007164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:40.022725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:40.023716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:40.024709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:40.024709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:40.024709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:40.026519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 452/886] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:40.035927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:40.131184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:40.147937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0df85c-6274-418a-a18b-d8146106d92f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:40.184396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:40.185013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.185385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.185657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.186300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:40.186622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:40.188048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:40.206406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 435/887] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:40.211674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:40.317731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbfc38-0eab-4724-8dc6-ec8c82593d9a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:40.335440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:40.355673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:40.356316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.356599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.356873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.365361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:40.366849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:40.379866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:40.380853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:40.381997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 453/888] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:40.389520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:40.495478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c21f1-3c32-4781-abd2-8bd460833787 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:40.531248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:40.532216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.532695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.533212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.534169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:40.534731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:40.536455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449c84a-bf17-4559-9e07-ebcfcd455658 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:40.562655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 436/889] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:40.564747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:40.678527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b570618-05b9-4e5f-baf7-dbcb8e92f176 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:40.719757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:40.720367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.720646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.720909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.721314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090841f9-3693-4e80-be0a-0f528c893cff None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.721595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090841f9-3693-4e80-be0a-0f528c893cff None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.721866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090841f9-3693-4e80-be0a-0f528c893cff None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.723584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 437/890] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:40.732264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367474a8-44d2-4dd7-b199-a8b216fb1b94 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.732879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367474a8-44d2-4dd7-b199-a8b216fb1b94 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.733369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367474a8-44d2-4dd7-b199-a8b216fb1b94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.740329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:40.762208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-367474a8-44d2-4dd7-b199-a8b216fb1b94 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:40.764777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-367474a8-44d2-4dd7-b199-a8b216fb1b94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:40.776680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:40.804353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:40.808527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:40.862374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 438/891] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:40 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:40.866057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d785d-cdff-4cbf-9520-67ebecae3c49 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.866327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d785d-cdff-4cbf-9520-67ebecae3c49 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.867044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d785d-cdff-4cbf-9520-67ebecae3c49 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.870214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 439/892] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:40.875648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:40.875925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:40.876189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:40.898777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:40.900374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:40.947427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 454/893] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:40 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:22:40.962593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:40.985401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 440/894] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:40 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:41.051092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.059854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d7017-31a9-4786-b2d1-c5a4daa2f91a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.095361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.096035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.096317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.096586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.106878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.108383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:41.123492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:41.124430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:41.125307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:41.125307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:41.125307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:41.129245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 455/895] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:41.131678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.202269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.246582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d36a7d-7824-4d69-b926-12ab4fb987d5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.277561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.281275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.281879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.282147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.282411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.283082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:41.283469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.284858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.298588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 441/896] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:41.315874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.420247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fa12e2-ddbe-4f87-92a4-17f8234e06df None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.449599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.452206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28023ab4-f5ee-4abb-ad62-fa14953e021f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.459308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.460378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.460660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.460930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.461514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.464253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:41.473544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:41.474574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:41.475434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 456/897] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:41.477262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.480551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.493554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.494585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.494843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.495082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.501974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.503530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:41.515053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:41.515979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:41.516845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 442/898] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:41.517318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.600322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dbddef-b35a-42a0-ba77-3c03ee939b8d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.632422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2042b44e-6f15-4c7d-add8-0ff0c23fa721 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.637574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.638196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.638471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.638778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.639443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.640863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.668639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 457/899] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:41 2022] POST /identity/v3/services => generated 241 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:41.668639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.698889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.699518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.699797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.700068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.700721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:41.701025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.702626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.713700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 443/900] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:41.715232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.740773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.793961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:41.803702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696f61a9-f807-41f6-ac1a-852352b59243 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.844252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.844953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.845232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.845506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.846246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:41.846585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.848235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.864299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 458/901] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:41.872384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:41.884228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbdd76-b477-4ac0-88c6-ebd9a7d72400 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:41.939057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:41.940135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:41.940641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:41.941147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:41.942087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:41.942387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:41.943863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:41.960307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 444/902] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:41.967582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:42.019675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24cab9-b77d-43f2-99d2-8ffb6e8c9c65 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:42.034698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:42.058435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:42.059071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.059347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.059613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.060336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] RBAC: Adding query filter params (role.id=951cca4e78364f3a8ea36c290df8e5f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=908ed003368445ae8db242549c212981) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:42.060636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:42.062059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7428eb2-7afe-4187-a7b2-11026fe5b4d1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:42.068027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 459/903] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/role_assignments?user.id=908ed003368445ae8db242549c212981&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=951cca4e78364f3a8ea36c290df8e5f3 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:42.069271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab35b74-28d9-4597-806c-b34c7e2eb7f4 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.069550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab35b74-28d9-4597-806c-b34c7e2eb7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.069816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab35b74-28d9-4597-806c-b34c7e2eb7f4 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.071298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 460/904] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:42.081071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.081497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.081866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.095876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6eb6e-c409-49a3-a220-0173684391c4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:42.106126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:42.109057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:42.111352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:42.160822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:42.163247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.163729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.164184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.165078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] RBAC: Adding query filter params (interface=public, service_id=31e0743f946e40959093b4657aa07c2f, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:42.165608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:42.167349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:42.176061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 445/905] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:41 2022] GET /identity/v3/endpoints?service_id=31e0743f946e40959093b4657aa07c2f&interface=public®ion_id=RegionOne => generated 194 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:42.182704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:42.221880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 461/906] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:42.311004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d4a39-7855-45c4-af9c-220f30a0bf90 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:42.347072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:42.347678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.347949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.348212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.348814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:42.350230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:42.361859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 446/907] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:42 2022] GET /identity/v3/services => generated 1497 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:22:42.506281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:42.619449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225e6255-6b3f-403d-8ca0-3552ee204ddd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:42.627959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:42.661451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:42.662120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.662394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.662695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.663327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:42.664724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:42.675497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:42.676495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:42.677384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 462/908] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Oct 4 08:22:42 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:42.684876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:42.726441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:42.842462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf95a92-b0c7-4bb8-92e2-a3e4cb82dbe1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:42.901474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:42.902466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:42.903004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:42.903458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:42.904537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:42.905050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:42.907297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:42.916051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 447/909] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:42 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:42.923904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:43.039915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32bc3bd-ebef-4e77-8a04-7af85c4e8836 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:43.075488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:43.076094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:43.076369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:43.076640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:43.077272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:43.077569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:43.078973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:43.084510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 463/910] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:42 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:43.091704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:43.213573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:43.217655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306c4e-bc39-4c8e-9360-049851e29a98 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:43.255610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:43.256216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:43.256490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:43.256757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:43.257454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] RBAC: Adding query filter params (interface=public, service_id=8a8d7fd0317247908a166e34dfbc2156, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:43.257757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:43.259712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f632afd0-47f1-40ef-a600-40bb3102bf30 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:43.266259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 448/911] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:43 2022] GET /identity/v3/endpoints?service_id=8a8d7fd0317247908a166e34dfbc2156&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:43.276477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:43.318237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:43.394959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68093618-7045-4c8d-aba3-9f9e0b86a1a8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:43.443935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:43.444992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:43.445488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:43.445952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:43.446835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:43.448556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3401c8-e447-4cb5-83cc-2777c3213fa5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:43.460960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 464/912] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:43 2022] GET /identity/v3/services => generated 1497 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:22:44.054694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacfbf60-941e-4f61-b8ca-dad8a058f09f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.055106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacfbf60-941e-4f61-b8ca-dad8a058f09f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.055288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacfbf60-941e-4f61-b8ca-dad8a058f09f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.056848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 449/913] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:44.064673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230eda7-4604-4500-b052-26bff6f779f4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.065638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230eda7-4604-4500-b052-26bff6f779f4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.066118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d230eda7-4604-4500-b052-26bff6f779f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.086219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d230eda7-4604-4500-b052-26bff6f779f4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:44.088360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d230eda7-4604-4500-b052-26bff6f779f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:44.179216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 465/914] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:44.459022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacfbf60-941e-4f61-b8ca-dad8a058f09f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:44.467847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9a4460-caa8-49f7-88a9-6f6bd3ab2692 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.468483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9a4460-caa8-49f7-88a9-6f6bd3ab2692 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.468955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9a4460-caa8-49f7-88a9-6f6bd3ab2692 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.470493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 466/915] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:44.475731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.476005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.476273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.499246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:44.500992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:44.532915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:44.606752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacfbf60-941e-4f61-b8ca-dad8a058f09f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:44.610616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 467/916] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:44.660539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:44.661177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.661446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.661710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.662296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:44.663677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-86e8254f-8902-4840-ba2f-e19f5060fb51 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:44.673246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:44.674243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:44.675115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:44.678670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 450/917] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:44.680697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:44.757495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67c685-0841-475b-8bde-5422783fe3ce None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.758104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67c685-0841-475b-8bde-5422783fe3ce None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.758608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67c685-0841-475b-8bde-5422783fe3ce None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.760585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 451/918] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:44.766972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.767405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.767743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.789339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:44.791618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:44.791919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2e077-6e8c-47e7-ac65-82ed9afc666a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:44.816571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:44.844175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:44.844835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:44.845113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:44.845387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:44.846472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:44.846820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:44.848752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:44.856546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 468/919] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:44.862445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:44.878686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 452/920] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:44.896023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:44.939469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:44.991339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4dd6ed-c7e2-48ba-ab54-da94d88ee1db None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.040087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.040962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.041365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.041470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.042105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.042446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.044140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.055946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 469/921] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:44 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:45.058173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.067395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa90ccf-d72c-4dfe-a843-70eed4a4be32 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.115008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.115647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.116067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.116354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.130651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:45.136442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:45.137425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:45.143576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 453/922] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:22:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:45.146210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.173899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.174958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.193489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58938101-1896-46e6-9611-73ea32fd9668 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.241724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.242339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.242652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.242925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.243670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] RBAC: Adding query filter params (interface=public, service_id=d8a51fca0034440f8b90125df7317502, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.243973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.245622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.255915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 470/923] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/endpoints?service_id=d8a51fca0034440f8b90125df7317502&interface=public®ion_id=RegionOne => generated 194 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:45.255915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.288436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadccade-337e-4653-aec1-aa2198698bd2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.323627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.324153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.324432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.324699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.325371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.325669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.327066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.340786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 454/924] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:45.340786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.355953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb5a6d-1368-4740-b12d-f0bbaf8f68ce None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.391761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.392391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.392670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.392942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.393523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.394931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:45.404775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:45.405811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:45.407328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 471/925] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:45.408109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.425927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.434327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.460070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bd4eb-473e-4ab4-93cd-19cd6f23a523 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.503957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.504693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.504977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.505247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.505860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.507372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.526587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 455/926] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/services => generated 1497 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:22:45.526587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.529434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673641cd-742a-4805-a882-9ecd63bde932 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.581599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.582281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.582779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.583117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.596697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.599187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:45.623866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:45.624891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:45.625783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:45.625783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:45.625783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:45.630053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 472/927] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:45.632328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.668955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda62d8-4bb1-449b-a72c-e89bd01d7778 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.716000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.716633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.717452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.718712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.719304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.720145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.720639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.722789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.729526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 456/928] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:45.735232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.746093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:45.763729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409652e-8452-455a-84e0-87fd8c8e5843 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.802062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.802985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.803411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.803827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.804644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.805162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.807056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ffad53-73b8-4645-821f-b5653caa7626 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.819682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 473/929] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:45.821238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd6781-c7a1-4844-95f3-ace576a64bab None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.822269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd6781-c7a1-4844-95f3-ace576a64bab None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.822969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd6781-c7a1-4844-95f3-ace576a64bab None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.824602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 474/930] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:45.828779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd6781-c7a1-4844-95f3-ace576a64bab None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:45.867403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcbac46-8833-4249-9401-87862dd06310 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.912396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:45.912396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.912396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.912396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.913103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:45.913103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:45.913103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c06410-2103-4563-ba12-1f34ef3daed6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:45.918334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 457/931] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:45.919508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:45.919789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:45.920061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:45.939464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:45.940668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:45.966233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd6781-c7a1-4844-95f3-ace576a64bab None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:45.994136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.006008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9392e457-f729-4372-beab-c7497055a829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.006789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9392e457-f729-4372-beab-c7497055a829 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.007166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9392e457-f729-4372-beab-c7497055a829 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.007442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9392e457-f729-4372-beab-c7497055a829 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.016079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9392e457-f729-4372-beab-c7497055a829 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.018034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.018593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9392e457-f729-4372-beab-c7497055a829 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.023743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 458/932] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:46.024479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9392e457-f729-4372-beab-c7497055a829 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:46.039625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:46.040870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:46.041788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:46.049140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 475/933] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:46.050343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9392e457-f729-4372-beab-c7497055a829 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.143979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b37725-8974-43b6-b948-53b1749929b6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.165909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9392e457-f729-4372-beab-c7497055a829 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.185473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.186384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.186703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.186981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.187655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.189025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.210639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.210639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.210639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.210639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.211338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:46.211338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.211338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.216138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 459/934] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:46 2022] POST /identity/v3/endpoints => generated 346 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:46.247090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 476/935] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:46.249924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.315909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.323068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.387169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.393315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770102a3-3a80-4802-8497-b2b905465274 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.440884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.441358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.441662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.441896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.459905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.461659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.464515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001270bc-51c0-4e25-adfc-68333e965da6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:46.483920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:46.485215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:46.486598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:46.486598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:46.486598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:46.487653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 460/936] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:46.496489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.514606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.514606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.514606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.514606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.515044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.515044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:46.524552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:46.525944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:46.527086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 477/937] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:46.533853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.640097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28dcd53-4429-40c5-bf5e-9ffb2881688b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.643364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.661232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb155ac-388b-4e8a-a87e-2f9717106daa None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.671997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.681921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.682606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.682962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.683237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.683985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:46.684289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.685711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.701909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 461/938] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:46.710582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.710743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.710743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.710743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.710743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.711075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:46.711075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.711961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.719558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 478/939] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:46.728170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.831516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054dd65-4052-454d-9684-67931aa5a4d7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.862068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b1de3f-74d7-4058-b5d0-38c948ac8e2f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:46.873866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.874814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.875256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.875667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.884348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.885900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.899399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:46.900904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:46.901467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:46.901951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:46.902428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:46.903337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:46.904421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 462/940] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:46.905641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:46.905956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:46.908213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:46.909455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.913587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 479/941] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:46.919532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:46.947392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:46.959263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:47.033015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333115b2-81f4-40a2-bb15-fa9ab2b13c86 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:47.077387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:47.078426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:47.078906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:47.079408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:47.080420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:47.080923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:47.082665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:47.084258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d587ee-90f9-4ff1-8b83-09f128901b9e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:47.105218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 463/942] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:47.107121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:47.134722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:47.135333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:47.135606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:47.135869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:47.136812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:47.138441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123ec9d5-6ff4-4481-9648-e73ac510fab5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:47.176132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 480/943] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:46 2022] POST /identity/v3/endpoints => generated 347 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:22:47.218380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:47.240095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a69f82c-dcef-47ea-a673-107da1b4d7e7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:47.290225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:47.292002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:47.292484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:47.293001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/908ed003368445ae8db242549c212981/roles/951cca4e78364f3a8ea36c290df8e5f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:47.295527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=908ed003368445ae8db242549c212981, role_id=951cca4e78364f3a8ea36c290df8e5f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:47.297206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd9618-e094-44c7-b433-68767587f7e3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:47.317676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 464/944] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:22:47 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/908ed003368445ae8db242549c212981/roles/951cca4e78364f3a8ea36c290df8e5f3 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:22:47.886332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14bb94-0af9-4dee-8577-fd3d5b4b4a6d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:47.886904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14bb94-0af9-4dee-8577-fd3d5b4b4a6d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:47.886904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f14bb94-0af9-4dee-8577-fd3d5b4b4a6d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:47.888447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 481/945] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:47.893894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:47.894295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:47.894464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:47.913477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:47.914675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:47.988732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 465/946] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:48.282425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f14bb94-0af9-4dee-8577-fd3d5b4b4a6d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:48.387263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:48.397465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f14bb94-0af9-4dee-8577-fd3d5b4b4a6d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:48.435351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:48.435904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.436176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.436441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.437019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:48.438387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:48.448254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:48.449270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:48.450095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 482/947] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:48.455399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:48.535445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:48.571407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bff6f-01e8-426e-8498-84dc3ae466fe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:48.611596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:48.612578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.613092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.613561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.614391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:48.614926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:48.616471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981c6568-39b3-46ed-b0d2-301f17da03d7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:48.623683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 466/948] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:48.634102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:48.639099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52d678-1727-4a50-b3f2-92811a677301 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.639584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52d678-1727-4a50-b3f2-92811a677301 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.639994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a52d678-1727-4a50-b3f2-92811a677301 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.641755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 467/949] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:48.647139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.647567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.648013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.668351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:48.670136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:48.751331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6521-38da-41a7-ab5a-a10ae4bd7627 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:48.753992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 468/950] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:48.792951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:48.794172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.794867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.795506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.796750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:48.797472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:48.800053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:48.807474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 483/951] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:48.813518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:48.881420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:48.919738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3911bc0-8218-4317-9cbd-0979513fa6e4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:48.954458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:48.955096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:48.955367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:48.955635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:48.956309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:48.957660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:48.980173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 469/952] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:48 2022] POST /identity/v3/endpoints => generated 349 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:22:49.062767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:49.089696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:49.202515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7947502-0e9c-4a46-866b-19e5da84d1e1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:49.247074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:49.248045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.248517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.248981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.249826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:49.251475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:49.254334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING keystone.server.flask.common [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 04 08:22:49.274370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.sql.core [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:22:49.274370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 08:22:49.274370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [parameters: {'id': '8475e7dc1a164347abbd582348f24526', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 08:22:49.274370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81405) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e0c791c1-253a-475a-9227-005cb8f2569a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8475e7dc1a164347abbd582348f24526.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8475e7dc1a164347abbd582348f24526. Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:22:49.293155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 04 08:22:49.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application yield resource Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise self.value Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application session.commit() Oct 04 08:22:49.295045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:22:49.295928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:22:49.296817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:22:49.297618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [parameters: {'id': '8475e7dc1a164347abbd582348f24526', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:49.298471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:49.301221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8475e7dc1a164347abbd582348f24526. Oct 04 08:22:49.303663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:49.307196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 484/953] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Oct 4 08:22:49 2022] POST /identity/v3/projects => generated 262 bytes in 245 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:49.314711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:49.434403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194da838-67ab-44dc-b916-6441970d0e34 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:49.452008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17699c-ac15-4d59-a4d1-c077793b2387 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.452319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17699c-ac15-4d59-a4d1-c077793b2387 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.452564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17699c-ac15-4d59-a4d1-c077793b2387 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.459358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:49.461354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 485/954] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:49 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:49.463507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd414-b1ad-4015-aa99-115935190721 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.463507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd414-b1ad-4015-aa99-115935190721 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.463967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4cd414-b1ad-4015-aa99-115935190721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.485960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7c4cd414-b1ad-4015-aa99-115935190721 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:49.487429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:49.487586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7c4cd414-b1ad-4015-aa99-115935190721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:49.488352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.488663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.489118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.500172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:49.502586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:49.520449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:49.521606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:49.523031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:49.526638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 470/955] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:49.531800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:49.587105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 486/956] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:49.670489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8d013-aa60-45e1-a199-c0fb3dbc3cdb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:49.706023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:49.707048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.707527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.707992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.708857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:49.709375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:49.711079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f261a0-db75-4bf1-84aa-be354a0df694 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:49.734036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 471/957] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:49 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:49.750958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b8284-607e-4de4-ba6d-8e68acca38fc None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.751305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b8284-607e-4de4-ba6d-8e68acca38fc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.751500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472b8284-607e-4de4-ba6d-8e68acca38fc None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.753203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 487/958] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:49.759114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:49.759861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:49.760337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:49.785165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:49.786484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:49.832835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:49.874569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 472/959] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:49.904553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472b8284-607e-4de4-ba6d-8e68acca38fc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:49.908327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:50.038014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472b8284-607e-4de4-ba6d-8e68acca38fc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.075726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.076359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.076616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.076882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.077497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.079021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:50.088716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:50.089892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:50.091050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 488/960] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:49 2022] GET /identity/v3/services/compute => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:50.098341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.158532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.227752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a7bde9-54d4-4a54-b96c-db73e90ea1fc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.262664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:50.271149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.271879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.272221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.272574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.273270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:50.273632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.276197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.282221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 473/961] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:50.287537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.311588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3182ef0-f976-4e60-8f07-096514027536 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.331518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:50.367834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.369153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.370287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.370593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.387119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:22:50.392753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:50.393720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:22:50.394589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:50.396955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 489/962] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:50.404859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.420201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7835065d-0c76-4905-ba17-3aac7880ad3c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.460505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.461345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.461466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.461743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.462440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] RBAC: Adding query filter params (type=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:50.463421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.464847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.478578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 474/963] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:50.542133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e49d1f-3e55-4a5c-89f0-d6e61844d7fd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.574436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:50.584428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.585055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.585353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.585627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.586315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:50.586647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.589011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.599032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 490/964] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:50.605398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.719957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:50.724072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe968ee-ba17-4238-b9e0-d03abad5b9fa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.770750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.771414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.771693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.772022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.785212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.786709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:50.802843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:50.803858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:50.805426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:50.805426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:50.805426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:50.805426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 475/965] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:50.810881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:50.924610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6572570b-25b9-4c92-8c89-df32ab1c00f0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:50.966576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:50.967567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:50.968071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:50.968542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:50.969434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:50.969936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:50.971597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:50.984997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 491/966] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:50.991677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.100666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e62fa85-5bf6-46bb-a7c5-e353b82269b4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:51.137488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:51.138131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.138403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.138703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.157720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:51.159190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:51.171489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:51.172434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:51.173844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 476/967] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:22:50 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:51.178403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.236399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:51.293904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a08d15-c182-49b3-972f-01b28aa8c589 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:51.335667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:51.336295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.336576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.336853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.337661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:51.337970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:51.339804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef774f-d103-44a8-8072-684504c445f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:51.361040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 492/968] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Oct 4 08:22:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:51.366784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.381313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:51.446056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d676dd-be67-4b18-9442-8a390f191acf None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.446409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d676dd-be67-4b18-9442-8a390f191acf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.446744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d676dd-be67-4b18-9442-8a390f191acf None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.448477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 493/969] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:51.453703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.453989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.454259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.473768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:51.475051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:51.487563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422dc1f4-c921-46bc-b6cd-e49050180c3d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:51.529252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:51.530388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.531352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.531858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.554896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 494/970] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:51.607289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:51.609097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:51.621893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:51.732959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 477/971] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:22:51 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:22:51.741194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.779334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:51.840797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.859877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd81f31-08ca-460e-acdc-042cfb66ffd3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:51.906843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:51.907487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:51.907763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:51.908031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:51.922683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:51.924159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:22:51.942371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:51.943666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:22:51.944549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:22:51.944549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:22:51.944549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:51.950886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 495/972] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:51.953213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:51.956993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb982d48-a19e-40ce-bcb9-457d4b56e442 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:51.971933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.008948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.010817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.011325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.011868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.012695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.014345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:52.025498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:52.026736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:52.027649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:52.035230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 478/973] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:22:52.036387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.063886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.081347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bdfd3d-58ab-44aa-ab77-7555bbf216ab None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.135292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.135902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.136179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.136448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.137104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.137423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.138831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da241bb-e710-44a9-91da-c022f0f77199 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.161356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a62e25-c242-4eb1-b171-6075f6b29dd4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.162810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 496/974] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:22:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:52.165049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798eb7d-c73d-42ac-b52a-89969e4e537e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.166188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798eb7d-c73d-42ac-b52a-89969e4e537e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.166869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798eb7d-c73d-42ac-b52a-89969e4e537e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.169090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 497/975] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:52.170470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798eb7d-c73d-42ac-b52a-89969e4e537e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.204079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.204685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.204961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.205228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.205852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.206144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.207561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f32631-ade4-492f-8771-2545b50bbfc8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.212832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 479/976] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:52.217829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.217829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.217829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.235395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:52.236824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:52.275945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.305057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798eb7d-c73d-42ac-b52a-89969e4e537e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.317990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 480/977] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:52.319438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.343733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.344366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.344643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.344968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.352298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.354456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:22:52.366270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:52.367214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.368093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 498/978] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:52.373504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.384901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.448156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97779d9-df90-4563-a0bc-ce567adbd443 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.485511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.486117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.486394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.486719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.487520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.487630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.488985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.494948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 481/979] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:52.500378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.510566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59566d97-d104-41da-9300-1f55e12afa3d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.535563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.563205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.564508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.565224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.565707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.567636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.568732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.572492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.600444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 499/980] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:52.602749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.614262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b5d83b-37c9-4fbf-9f03-fa59fcd83943 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.665023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.665699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.665978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.666432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.667470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] RBAC: Adding query filter params (interface=admin, service_id=d8a51fca0034440f8b90125df7317502, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.667470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.668946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.675066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 482/981] 10.222.0.16 () {56 vars in 1249 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/endpoints?service_id=d8a51fca0034440f8b90125df7317502&interface=admin®ion_id=RegionOne => generated 193 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:52.675738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.685018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.730279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf5b7d-8730-4f0e-8d1d-b9057062eb00 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.774121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.775457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.775848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.776152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.776801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.778229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.789788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.793373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.sql.core [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:22:52.793373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 08:22:52.793373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [parameters: {'user_id': '6f60b83e728246ffb8de14d04f347bc3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 08:22:52.793373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81405) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 04 08:22:52.797828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d8734-d573-4c30-a1c2-75b37ac3d389 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Oct 04 08:22:52.800313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:22:52.801239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application yield resource Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.rollback() Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.force_reraise() Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise self.value Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application session.commit() Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 04 08:22:52.802215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._prepare_impl() Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.flush() Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._flush(objects) Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application compat.raise_( Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application flush_context.execute() Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rec.execute(self) Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 04 08:22:52.803169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 04 08:22:52.804086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 04 08:22:52.804951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [parameters: {'user_id': '6f60b83e728246ffb8de14d04f347bc3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:52.805804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 04 08:22:52.807216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 500/982] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Oct 4 08:22:52 2022] POST /identity/v3/users => generated 160 bytes in 200 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:52.808157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.839653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.840175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.840449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.840732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.841440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] RBAC: Adding query filter params (role.id=951cca4e78364f3a8ea36c290df8e5f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=908ed003368445ae8db242549c212981) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:52.841738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.843258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299484f9-68ba-469b-bd5b-bf2b74856242 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.850013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 483/983] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/role_assignments?user.id=908ed003368445ae8db242549c212981&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=951cca4e78364f3a8ea36c290df8e5f3 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:52.851242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb82415-355a-4735-9fc9-19d85d399867 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.851726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb82415-355a-4735-9fc9-19d85d399867 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.852181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb82415-355a-4735-9fc9-19d85d399867 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.855372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 484/984] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:52.856543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb82415-355a-4735-9fc9-19d85d399867 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:52.937325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707bdf8-0499-4a93-8089-1d22783cecac None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:52.971235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:52.981848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:52.982513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.982821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.983085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:52.983750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:52.985321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b92c7-b4be-4ef4-b592-d3b7ea1cd010 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:52.995413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 501/985] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/services => generated 1497 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:22:52.997464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:52.997743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:52.998012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:53.012900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb82415-355a-4735-9fc9-19d85d399867 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:53.018987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:53.020697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:53.059039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:53.059925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:53.060370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:53.061539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:53.071948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:53.072572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:22:53.087954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:53.089155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:53.090125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 485/986] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:22:52 2022] GET /identity/v3/users/cinder => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:53.094797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:53.111902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 502/987] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:53.120923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:53.233804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7271bde-64bd-4d3b-8a89-ec4034e6d145 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:53.284400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:53.285164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:53.285530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:53.285803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:53.286515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:53.286845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:53.288352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2da8d0d-9c84-4d5c-af02-7e7f0ae31379 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:53.329115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 486/988] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Oct 4 08:22:53 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:53.391783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:53.505343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e368173b-3733-4d6d-9348-01df66d95f12 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:53.532013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:53.543885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:53.544494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:53.544767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:53.545032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:53.545697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:53.547087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa197600-b80b-4d6e-b70b-219c3379a650 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:53.563861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 503/989] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:53 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:55.249469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a8cff-0327-4818-a742-4c5dcd6592d8 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.249982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a8cff-0327-4818-a742-4c5dcd6592d8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.249982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a8cff-0327-4818-a742-4c5dcd6592d8 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.251564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 487/990] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:55.253620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a393fa-04a3-4ff6-8a4d-7a08731e50c3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.254057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a393fa-04a3-4ff6-8a4d-7a08731e50c3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.254257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a393fa-04a3-4ff6-8a4d-7a08731e50c3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.255713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 504/991] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:55.256494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.256787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.257031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.261677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.261957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.262244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.278679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:55.280471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:55.291568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:55.293344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:55.372810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 488/992] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:55.387528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 505/993] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:55.641335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:55.645145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:55.656012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:55.760564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba824cc7-e468-457f-ad3e-ee87c05cd629 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:55.784726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f220b1e7-b789-4650-b36e-1cf889acac3c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:55.804349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:55.804982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.805254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.805519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.806192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:55.807244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:55.808668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:55.820569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:55.821724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:22:55.822731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 489/994] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Oct 4 08:22:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:55.825045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:55.837095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:55.837941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.838283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.838730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] PATH_INFO: `/v3/services/image` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.839121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:55.840505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:55.849899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:55.851897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:55.854042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 506/995] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Oct 4 08:22:55 2022] GET /identity/v3/services/image => generated 86 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:55.858067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:55.905041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:55.939553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0456cd3-b478-4d87-9585-72723e717522 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:55.968358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fa125-4af0-47b3-ae7c-df13b1b71a15 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:55.974168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:55.974809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.975088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.975362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.975992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:55.976297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:55.978758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c313a6-05ec-4d01-aaae-f56cb0cc34a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:55.984143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 490/996] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:55.985125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc478158-5bac-4eb3-8167-a4086730ab84 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:55.985401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc478158-5bac-4eb3-8167-a4086730ab84 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:55.985656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc478158-5bac-4eb3-8167-a4086730ab84 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:55.986890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 491/997] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:55.989501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc478158-5bac-4eb3-8167-a4086730ab84 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.009378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.010013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.010292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.010574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.011213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] RBAC: Adding query filter params (name=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:56.011511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.012878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b4b25-1da2-4deb-b117-c373bcfe2627 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.017581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 507/998] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:56.018873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2183665d-4481-4764-976c-25d66b938637 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.019154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2183665d-4481-4764-976c-25d66b938637 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.019422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2183665d-4481-4764-976c-25d66b938637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.037910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2183665d-4481-4764-976c-25d66b938637 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:56.039123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2183665d-4481-4764-976c-25d66b938637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:56.079900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:56.098943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc478158-5bac-4eb3-8167-a4086730ab84 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.113968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 508/999] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:56 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:56.114838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2183665d-4481-4764-976c-25d66b938637 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.146511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:56.152950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.154095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.154590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.155122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.156182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:56.156708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.159278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.176617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 492/1000] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Oct 4 08:22:55 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:56.177893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.246229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2183665d-4481-4764-976c-25d66b938637 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.282752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.283383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.283682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.283960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.284179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2850d227-484e-4db5-b506-52b68ea6892c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.284601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] RBAC: Adding query filter params (type=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:56.284938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.286314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.291314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 509/1001] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:56 2022] GET /identity/v3/services?type=image => generated 122 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:56.323254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.323515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.323599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.323908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.324531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.325938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.350604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 493/1002] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:22:56 2022] POST /identity/v3/endpoints => generated 325 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:22:56.390197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.438421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:56.523278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145eb5cc-bd8c-4b24-b056-d62fbd728d6e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.568519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.569151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.570095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.570095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.570293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.571644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:22:56.582180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:22:56.583266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:22:56.584757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 510/1003] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:56 2022] GET /identity/v3/services/compute => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:22:56.590231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.647130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:56.694362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e727a76-212b-4c9d-b254-2efb4b3ca076 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.730638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.731249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.731524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.731790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.732415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:56.732714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.734071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.739257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 494/1004] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:56 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:56.744968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:56.861939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:56.872740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb3b94-a1b6-4bb1-b3ef-f67533874b93 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:56.924101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:56.924709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:56.925045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:56.925327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:56.925980 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] RBAC: Adding query filter params (type=compute) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:56.926283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:56.927691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:56.933813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 511/1005] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:22:56 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:22:56.939446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:57.053690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5644a02-a03a-477c-a5ef-3018fcbd6b14 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:57.076833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:57.096091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:57.096713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:57.096986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:57.097253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:57.097955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] RBAC: Adding query filter params (interface=public, service_id=15face6fdb0c4cb08827c3667b960efa, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:22:57.098257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:57.099632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf47b07f-2aa7-40b1-becb-213b6e199a0c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:57.105562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 495/1006] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:22:56 2022] GET /identity/v3/endpoints?service_id=15face6fdb0c4cb08827c3667b960efa&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:57.111025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:57.245437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e93205d-243b-4e01-a0ce-dea3c336d4de None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:57.285409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:57.286436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:57.286436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:57.287008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:57.287257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:57.288618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f409d0-7b71-4f8f-a914-1a2df205fe50 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:57.302586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 512/1007] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:22:57 2022] GET /identity/v3/services => generated 1727 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:22:58.552881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b6322b-122c-4a2d-96f8-2a5352145d7a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:58.553388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b6322b-122c-4a2d-96f8-2a5352145d7a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:58.553388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b6322b-122c-4a2d-96f8-2a5352145d7a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:58.554736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 496/1008] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:58.560656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:58.561117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:58.561196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:58.580800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:58.581982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:58.602289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:58.621240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d16140-92d5-4620-8a04-3c2c29d42833 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:58.621588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d16140-92d5-4620-8a04-3c2c29d42833 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:58.621813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d16140-92d5-4620-8a04-3c2c29d42833 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:58.623176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 497/1009] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:22:58.628466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:58.628739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:58.629010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:58.649552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:58.650781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:58.656489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 513/1010] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:58 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:58.717178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 498/1011] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:58 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:22:58.927893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:58.990419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:59.032208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862bf3e0-d021-475d-a59d-d86fb1ac07f1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:59.067797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:59.068394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:59.068669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:59.068936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:59.069595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:59.070991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:59.089208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 514/1012] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:58 2022] POST /identity/v3/services => generated 242 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:59.104926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3be994-7a21-48fa-ab0c-f31da7ad83a0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:59.123755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:59.161789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:59.162519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:59.162982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:59.163249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:59.164517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:59.166101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be48e4c-3ee9-4034-b6cd-0022cb867ec3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:59.167953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 499/1013] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:22:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:22:59.173689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:59.192058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:59.285604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1532efd-41c2-447b-bdbf-54f2588a3c0b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:22:59.319717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:22:59.320321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:59.320600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:59.320866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:59.321519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:22:59.323636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e6a076-d652-4822-8a10-fa949b4fa2f4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:22:59.351336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 515/1014] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:22:59 2022] POST /identity/v3/projects => generated 311 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:22:59.504370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cbcf1-dbfa-4898-957e-cc5d342a769d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:59.504566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cbcf1-dbfa-4898-957e-cc5d342a769d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:59.504823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cbcf1-dbfa-4898-957e-cc5d342a769d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:59.506335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 500/1015] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:22:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:22:59.512040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:22:59.512331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:22:59.512594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:22:59.531012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:22:59.532210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:22:59.594985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:22:59.606588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 516/1016] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:22:59 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:22:59.884387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94cbcf1-dbfa-4898-957e-cc5d342a769d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:22:59.984677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94cbcf1-dbfa-4898-957e-cc5d342a769d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:00.020986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:00.021615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:00.021890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:00.022157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:00.022760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:00.024105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:00.039488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:00.040510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 04 08:23:00.041498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:00.045485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 501/1017] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:22:59 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:00.046769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:00.158245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6364160-6cf8-4ea1-8c77-34804f7e3802 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:00.194304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55270d44-33c5-434f-9056-663e0383030c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:00.194940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55270d44-33c5-434f-9056-663e0383030c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:00.195218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55270d44-33c5-434f-9056-663e0383030c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:00.195489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55270d44-33c5-434f-9056-663e0383030c None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:00.196120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55270d44-33c5-434f-9056-663e0383030c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:00.196423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55270d44-33c5-434f-9056-663e0383030c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:00.197938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55270d44-33c5-434f-9056-663e0383030c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:00.207585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 517/1018] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:23:00 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:00.208553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:00.252943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:00.313421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbace7c-3bd5-46e7-bc99-212892b8a611 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:00.349937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:00.350566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:00.350853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:00.351122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:00.351774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:00.352074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:00.353977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0209ae6-8960-4084-9d50-2c5a5ef01498 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:00.359130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 502/1019] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Oct 4 08:23:00 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:00.364886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55270d44-33c5-434f-9056-663e0383030c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:00.424914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:00.465642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55270d44-33c5-434f-9056-663e0383030c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:00.499317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:00.499915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:00.500188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:00.500452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:00.501132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:00.502484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59f5e9a-948b-4b2f-810c-db65dd59bc80 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:00.530585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 518/1020] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:23:00 2022] POST /identity/v3/endpoints => generated 334 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:01.309755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a5340-640a-40ae-bd6c-cbe50837e73a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:01.310271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a5340-640a-40ae-bd6c-cbe50837e73a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:01.310379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455a5340-640a-40ae-bd6c-cbe50837e73a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:01.311780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 503/1021] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:01.317390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c5666-7b76-44e7-8162-65d2926668ab None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:01.317858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c5666-7b76-44e7-8162-65d2926668ab None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:01.317935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9c5666-7b76-44e7-8162-65d2926668ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:01.336636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7e9c5666-7b76-44e7-8162-65d2926668ab None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:01.337843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7e9c5666-7b76-44e7-8162-65d2926668ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:01.414376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 519/1022] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:01.588106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d0157-6d78-44e3-b620-54f68b441d94 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:01.588477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d0157-6d78-44e3-b620-54f68b441d94 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:01.588715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d0157-6d78-44e3-b620-54f68b441d94 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:01.590133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 504/1023] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:01.595436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:01.595720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:01.595990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:01.613776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:01.614970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:01.652280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:01.688882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d0157-6d78-44e3-b620-54f68b441d94 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:01.694596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 520/1024] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:01.809697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d0157-6d78-44e3-b620-54f68b441d94 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:01.816600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:01.844409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:01.845026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:01.845297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:01.845558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] PATH_INFO: `/v3/services/image` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:01.846124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:01.847489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:01.857076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:01.858115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:01.859513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:01.866390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 505/1025] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Oct 4 08:23:01 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:01.866526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:01.960973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:01.967017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c341f0-4c49-40e9-8ab8-0c5d03a37eba None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.006284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.008605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.008605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.008605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.008605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] RBAC: Adding query filter params (name=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:02.008605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.009735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.021185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 521/1026] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:01 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:02.023643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:02.059870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:02.092951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e0ac6-8efb-4494-9e3a-1d5c6d09d6ac None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.132875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072e19a-c9a6-41a2-b56a-5ca3469d291e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.147411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.148409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.149186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.149372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.154109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.155679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.162520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 506/1027] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:02.163216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:02.180720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.181361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.181827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.182261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.182936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] RBAC: Adding query filter params (type=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:02.183235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.184659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.189489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 522/1028] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:02 2022] GET /identity/v3/services?type=image => generated 350 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:02.195237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:02.214259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:02.290500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb985ab-ddf5-4064-b0f9-d63b46e2121c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.305483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:02.309845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1161f20-b3d4-4ca2-9c57-1c170e8efc83 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.327471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.328222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.328347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.328595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.329250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.330720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.349861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.350759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.351135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.351367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.352074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] RBAC: Adding query filter params (interface=public, service_id=c0a2f7b8d1204434851dc1c7768ea97d, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:02.352375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.354497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e2a71-873a-4d36-a0bd-cf8d769f172d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.359331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 507/1029] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:23:02 2022] POST /identity/v3/users => generated 285 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:02.359846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 523/1030] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:23:02 2022] GET /identity/v3/endpoints?service_id=c0a2f7b8d1204434851dc1c7768ea97d&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:02.365529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:02.493290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:02.503354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9594ff5e-5b54-40af-a4f1-fab1e3661773 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:02.545433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:02.546059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.546336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.546637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.547253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:02.548617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e70c479-11d1-43eb-b5a5-44a0e2c8ae2f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:02.555684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 508/1031] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:23:02 2022] GET /identity/v3/services => generated 1957 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:23:02.863671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332c66b-3050-4632-ac34-9e490a343d9f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.863932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332c66b-3050-4632-ac34-9e490a343d9f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.864256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7332c66b-3050-4632-ac34-9e490a343d9f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.865765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 524/1032] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:02.872952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:02.873243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:02.873520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:02.892899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:02.894085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:02.970573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 509/1033] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:02 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:03.249161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7332c66b-3050-4632-ac34-9e490a343d9f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:03.362857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7332c66b-3050-4632-ac34-9e490a343d9f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:03.399258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:03.399885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:03.400734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:03.401005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:03.411650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:03.416502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:03.417547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:03.418534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:03.425938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 525/1034] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:23:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:03.426077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:03.533937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:03.548036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a2b902-86bd-4ac3-a846-4c35f4fd0168 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:03.583001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:03.583604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:03.583882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:03.584153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:03.584812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:03.585114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:03.586465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:03.599066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 510/1035] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:23:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:03.600469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:03.627493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:03.709503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd27f59-e487-4e43-9630-37857f755c9e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:03.746757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:03.747374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:03.747647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:03.747915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:03.759356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:03.761569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:03.776933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:03.778034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:03.778977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:03.778977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:03.778977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:03.782942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 526/1036] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:03.783801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:03.886733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a60b7-fee4-4bb8-b55a-5653767d9973 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:03.920668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:03.921270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:03.921542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:03.921806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:03.922437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:03.922762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:03.924112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:03.940671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 511/1037] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:03.942489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:04.020968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:04.049642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1aeee04-52a4-4385-ab0d-ce05284bb4a2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:04.084159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:04.084790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.085067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.085342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.106229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:04.108792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:23:04.123001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:04.124031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:04.129872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 527/1038] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:23:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:04.132386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:04.147034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:04.239664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bf5b19-de12-4f46-8c8f-5e74ee9cdd6e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:04.275599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:04.276158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.276422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.276679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.277389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:04.277686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:04.279191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6360aea9-efe3-4240-9227-e5a09e340aa0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:04.301592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 512/1039] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:23:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:04.303277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:04.420460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed416a-5a9a-4622-98f6-8c3ae479e801 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:04.461542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:04.463129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.463603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.463603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.470469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:04.554879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:04.556293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:04.617196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f20f317-cb01-44d7-875c-16c0db369c28 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.617739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f20f317-cb01-44d7-875c-16c0db369c28 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.617739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f20f317-cb01-44d7-875c-16c0db369c28 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.619505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 513/1040] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:04.624825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730dc0-2a87-40b4-a7fc-46893a401540 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.625100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730dc0-2a87-40b4-a7fc-46893a401540 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.625369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3730dc0-2a87-40b4-a7fc-46893a401540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.643698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d3730dc0-2a87-40b4-a7fc-46893a401540 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:04.644873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d3730dc0-2a87-40b4-a7fc-46893a401540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:04.660612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 528/1041] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:04 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:04.662602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:04.703403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:04.724972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:04.743170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 514/1042] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:04 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:04.796552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16bb3db-4e7a-49bd-aa5f-0dcf187c9f2f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:04.840411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:04.841125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.841514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.841869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.855799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:04.857647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:04.865762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ede53b2-7ca2-4e5a-bdea-67c2f78e8f39 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.870886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ede53b2-7ca2-4e5a-bdea-67c2f78e8f39 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.870886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ede53b2-7ca2-4e5a-bdea-67c2f78e8f39 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.870886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 515/1043] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:04.875037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:04.875246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:04.875525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:04.876776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:04.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:04.878957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:04.878957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:04.878957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:04.882745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 529/1044] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:04.883605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:04.896872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:04.898094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:04.971110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 516/1045] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:04 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:04.990619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.006987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d01ef5-dc6f-4436-9a46-36071b0bdc73 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.022008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.047262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.047862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.048144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.048419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.049063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.049355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.050760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.064318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 530/1046] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:05.069898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.139305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e419b8-f3a8-44af-b203-abc2fa238817 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.179808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.192680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.193406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.193752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.194097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.195012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36428a87-1466-4541-9391-f9bfeaef9349 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.206292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:05.211212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:05.212359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:05.213947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:05.215247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 517/1047] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:05.222245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.249867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.250638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.250920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.251192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.258478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.260014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:23:05.271406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:05.280582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 531/1048] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:05.282225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.286850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.343478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee43a1b6-46b4-4be2-a10b-1e4628fc29fb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.383369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.384099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.384944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.385048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e024ab41-a6ad-43d6-91cf-3fbd0021adf0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.385137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.386860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.387161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.388536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.396923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 518/1049] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:05.397645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.426069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.426907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.427130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.427404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] PATH_INFO: `/v3/services/image` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.427990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.429401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:05.438309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:05.439432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:05.440611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 532/1050] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/services/image => generated 86 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:05.442469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.459569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.521440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdddfe2-2743-4f84-b2c5-124605121786 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.525759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.559435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d07393f-0310-47f6-bf0c-5cc86db75db7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.566475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.567396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.568012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.568287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.568985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.569291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.570879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.588804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 519/1051] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:05.589664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.601914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.602574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.602856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.603215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] PATH_INFO: `/v3/users/06c40a9b2ec54b608991e3a8a90625b9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.617803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] RBAC: Authorizing `identity:get_user(user_id=06c40a9b2ec54b608991e3a8a90625b9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.619516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.622938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 533/1052] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/users/06c40a9b2ec54b608991e3a8a90625b9 => generated 285 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:05.622938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.716002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a9d7b2-6806-44f8-8f90-e0c94fcf4b4e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.729815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.747598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fa774c-5853-4e23-8df5-4af4591dbf08 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.761728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.762681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.762952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.763218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.764068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] RBAC: Adding query filter params (name=image) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.764482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.765924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.777264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 520/1053] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/services?name=image => generated 122 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:05.777264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.790870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:05.800616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.801334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.801650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.801974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.802824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] RBAC: Adding query filter params (role.id=951cca4e78364f3a8ea36c290df8e5f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.803216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.804756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.813191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 534/1054] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=951cca4e78364f3a8ea36c290df8e5f3 => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:05.814093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:05.902171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06bd92-b594-45ea-8c1c-5b84afe3c51b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.943345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.944020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.944303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.944594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.948190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a850a-1f2e-446f-9b87-83ba02658ae0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:05.992093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:05.993137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:05.993567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:05.994586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:05.995629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] RBAC: Adding query filter params (type=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:05.995956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:05.998092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:05.998531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:06.009139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 535/1055] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/services?type=image => generated 350 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:06.017082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:06.030784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:06.032416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:06.086724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:06.135222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0c3c60-7b83-4dd8-9905-ff032609f703 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:06.153899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 521/1056] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:05 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:06.156403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:06.176748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:06.177523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:06.177935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:06.178283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:06.179080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:06.180520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:06.209385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 536/1057] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:23:06 2022] POST /identity/v3/endpoints => generated 327 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:06.254516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:06.295036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3794b54-a610-4461-b400-a2b0681bffb9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:06.344938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:06.345958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:06.346384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:06.346854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:06.348226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:06.349973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d17003-829e-4875-89d8-42bda22023d0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:06.352315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 522/1058] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:06 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:06.359324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:06.476024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c72987-a0f3-4da2-a4c0-1e80ee1cc64c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:06.491212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:06.514681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:06.515207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:06.515479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:06.515744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:06.516469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=5738d9ca67834a42b22fead73a050b99, user.id=06c40a9b2ec54b608991e3a8a90625b9) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:06.516770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:06.518193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20b20c0-3321-432c-ae99-41c664e6c23c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:06.524399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 537/1059] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:06 2022] GET /identity/v3/role_assignments?user.id=06c40a9b2ec54b608991e3a8a90625b9&scope.project.id=5738d9ca67834a42b22fead73a050b99&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:08.008447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60defca8-193b-46e3-ba98-885d9a10e165 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.008985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60defca8-193b-46e3-ba98-885d9a10e165 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.008985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60defca8-193b-46e3-ba98-885d9a10e165 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.010960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 523/1060] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:08.016227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb55d88-aa51-410b-ae21-1d0128112540 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.016637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb55d88-aa51-410b-ae21-1d0128112540 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.016818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb55d88-aa51-410b-ae21-1d0128112540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.035584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1cb55d88-aa51-410b-ae21-1d0128112540 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:08.036753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1cb55d88-aa51-410b-ae21-1d0128112540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:08.104050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 538/1061] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:08.371351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60defca8-193b-46e3-ba98-885d9a10e165 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:08.469380 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df9bb-7da7-4b76-883f-263b7dfab101 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.469746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df9bb-7da7-4b76-883f-263b7dfab101 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.469943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df9bb-7da7-4b76-883f-263b7dfab101 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.471325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 539/1062] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:08.476472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.476763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.477041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.481403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60defca8-193b-46e3-ba98-885d9a10e165 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:08.496503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:08.497740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:08.521430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:08.522114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.522381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.522680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.530911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:08.538188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:08.543082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:08.544237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:08.545093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:08.546824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 524/1063] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:23:08 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:08.550337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:08.584540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 540/1064] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:08.679139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d55cd1d-db4d-478b-b250-18eee2f437eb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:08.714436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:08.715073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.715351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.715621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.716283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:08.716583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:08.717948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4450c5-f1ca-48a0-88d2-c7acf0069543 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:08.726520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 525/1065] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:23:08 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:08.732945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:08.776964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5516c-a43f-476d-b33e-71dafbfc735c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.777315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5516c-a43f-476d-b33e-71dafbfc735c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.777512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5516c-a43f-476d-b33e-71dafbfc735c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.779148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 526/1066] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:08.779393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:08.784258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.784535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.784808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.804345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:08.805535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:08.848553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a693f55e-2f1d-4c8c-b6a8-ceee9b63bfd0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:08.869448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:08.889072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 527/1067] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:08.890003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:08.894980 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:08.895851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:08.896312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:08.896788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:08.907542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:08.909366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:08.925393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:08.926439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:08.927452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:08.927452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:08.927452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:08.927452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 541/1068] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:08.932968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.032470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8044e77-6226-4bbd-a294-1b6e52b1417f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.051601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.061840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6201ad8b-5a79-4fee-a605-dcee9203c364 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.068333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.068944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.069212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.069696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.102798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.103447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.103726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.103995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.104632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:09.104945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.107355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.126373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 542/1069] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:09.127798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.143482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.157213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.158691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.240125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3518731-d063-49df-a8f6-2337da09b3d9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.263417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 528/1070] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:09.263417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.277374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.278009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.278287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.278567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.285827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.287310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:23:09.299559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:09.300548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 543/1071] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:09.301526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.319173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.372973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.384093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347d98de-18eb-4444-9db8-b1b3dbcda3a7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.419614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ad83b-cda0-4083-a7af-ecb4b7e07dc2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.424820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.425442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.425714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.425976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.437078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:09.442122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:09.443102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:09.443933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 529/1072] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:09.444641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.460897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.461519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.461795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.462058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.473453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.474958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:09.489792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:09.491041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:09.492398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:09.492398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:09.492398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:09.492398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 544/1073] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:09.492398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.550871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9e2107-b13a-4004-92d1-a49d10f87f82 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.559916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.588393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.589071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.589351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.589622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.590353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:09.590792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.592209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.599137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91854537-c5ac-4054-9584-1d696e60ee29 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.610135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 530/1074] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:09.611088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.621546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.635839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.636455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.636734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.637011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.637683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:09.637985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.639397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.655402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 545/1075] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:09.655402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.717388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2ccd7-eccc-4df5-8460-7e6ef810ef17 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.758144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f35c-2719-43b5-b896-f20c1cd5b2ae None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.758393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.758962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.759440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.759880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.760798 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:09.761302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.763267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.776541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 531/1076] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:09.777374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.788274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.805240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.805884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.806230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.806515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.824814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.827523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:09.855554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:09.856903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:09.858004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:09.858004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:09.858004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:09.858004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 546/1077] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:09.864589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.876667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:09.921126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781c7fa9-1fe3-45f9-a7be-b4ea54e39440 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:09.961289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:09.961925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:09.962204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:09.962499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] PATH_INFO: `/v3/users/06c40a9b2ec54b608991e3a8a90625b9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:09.964552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] RBAC: Authorizing `identity:get_user(user_id=06c40a9b2ec54b608991e3a8a90625b9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:09.965985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:09.967647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 532/1078] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/users/06c40a9b2ec54b608991e3a8a90625b9 => generated 285 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:09.973646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:09.983343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b885790-215f-488b-9751-b054fa440ff8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.019873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.020414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.020694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.021328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.021991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:10.022693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.024103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.039996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 547/1079] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:10.045985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:10.086613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:10.104639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c65d83-642c-4811-8bda-8f638760bce1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.145432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.146070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.146353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.146651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.147773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.149206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.154758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 533/1080] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:09 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:10.157060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:10.161191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:10.163589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25240a0-e368-4fd4-9ac0-e20dd4c4a698 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.199914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.200547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.200826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.201097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.207877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.209338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:23:10.220671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:10.221657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:10.222589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 548/1081] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:23:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:10.227678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:10.258713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e27141-b0b7-4309-b523-ff1ac804f3aa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.293150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.293726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.294001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.294271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99/users/06c40a9b2ec54b608991e3a8a90625b9/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.296236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] RBAC: Authorizing `identity:create_grant(project_id=5738d9ca67834a42b22fead73a050b99, user_id=06c40a9b2ec54b608991e3a8a90625b9, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.297662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.311552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 534/1082] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:23:10 2022] PUT /identity/v3/projects/5738d9ca67834a42b22fead73a050b99/users/06c40a9b2ec54b608991e3a8a90625b9/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:23:10.334730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:10.343598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28218a2-959b-44ac-b2a0-670a50229b17 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.388713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.389550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.390017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.390468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.391764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:10.392304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.395028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca0c10d-f5a4-429d-baed-b98e583f651f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.410195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 549/1083] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:23:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:10.417172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:10.482925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:10.523820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb532e-352d-487c-a4c5-9613993ca1c7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:10.558516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:10.559124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:10.559398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:10.559667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/951cca4e78364f3a8ea36c290df8e5f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:10.561796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=02d91fccc1c8400392fbbc34416f552b, role_id=951cca4e78364f3a8ea36c290df8e5f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:10.563253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5872f-e4ff-47bb-beb8-145d9b305c87 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:10.576005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 535/1084] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:23:10 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/02d91fccc1c8400392fbbc34416f552b/roles/951cca4e78364f3a8ea36c290df8e5f3 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:23:12.521201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfde65e-209e-493c-a63e-a5ed21f62db4 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:12.521703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfde65e-209e-493c-a63e-a5ed21f62db4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:12.521703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfde65e-209e-493c-a63e-a5ed21f62db4 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:12.523186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 550/1085] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:12.529186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f98c-aaf3-48b8-8999-ae01a0447491 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:12.529590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f98c-aaf3-48b8-8999-ae01a0447491 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:12.529725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f98c-aaf3-48b8-8999-ae01a0447491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:12.548493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d474f98c-aaf3-48b8-8999-ae01a0447491 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:12.549672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d474f98c-aaf3-48b8-8999-ae01a0447491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:12.616734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 536/1086] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:12.793016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e736c026-9e83-4f3a-85c8-d990a13ef87c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:12.793195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e736c026-9e83-4f3a-85c8-d990a13ef87c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:12.793464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e736c026-9e83-4f3a-85c8-d990a13ef87c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:12.794840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 551/1087] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:12.800046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:12.800330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:12.800606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:12.819177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:12.820362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:12.871773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:12.890821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 537/1088] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:12.913178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e736c026-9e83-4f3a-85c8-d990a13ef87c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.018293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e736c026-9e83-4f3a-85c8-d990a13ef87c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.053417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.053960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.054234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.054498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.065736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:13.070409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:13.071521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:13.073305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 552/1089] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:13.079480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.165058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.185643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.222079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f774a4-926a-4078-bfe0-e07e9a7b9fdc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.255306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.255911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.256190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.256459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.257115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:13.257415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.258812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.267652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 538/1090] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:13.271759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30112845-863b-4cd6-ac42-4627771d0387 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.273697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.308707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.309256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.309528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.309795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.320157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:13.321450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:13.326120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:13.327133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:13.328140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 553/1091] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:13.335579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.379621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d911a2f-d7f7-4316-a7a5-840a646e3215 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.413549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.415636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.416278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.416555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.416829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] PATH_INFO: `/v3/users/06c40a9b2ec54b608991e3a8a90625b9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.418173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] RBAC: Authorizing `identity:get_user(user_id=06c40a9b2ec54b608991e3a8a90625b9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.419646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.426150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 539/1092] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/users/06c40a9b2ec54b608991e3a8a90625b9 => generated 285 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:13.427075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.444268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3042fa-2f88-4e82-b00e-96d68ff5a3c3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.477292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.477900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.478183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.478458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.479239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:13.479552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.480934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.493642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 554/1093] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:13.500266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.536617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779fc1dd-7a28-4a63-ad34-960e8990bfb6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.550917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.573847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.574378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.574684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.574954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.610671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eda5862-db64-474c-b5dc-28580f2db7b0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.644828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.645594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.645855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.645971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.646122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.647319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.653889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.658780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.660276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:13.674229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:13.675222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:13.676171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:13.676171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:13.676171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:13.676171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 555/1094] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:23:13.681139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.751057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 540/1095] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 04 08:23:13.757169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.771722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.791871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69570-6af5-47dd-8a6a-245d0ddd4afc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.825622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.826164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.826436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.826745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.827387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:13.827685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.829070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.841882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 556/1096] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:13.847786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.866725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbbe975-e004-4c4d-a846-9fcd00e9602f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.898707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:13.901888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.902524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.902831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.903106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:13.904230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:13.905651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:13.907338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 541/1097] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:13.913375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:13.962149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29582574-6c9e-4059-8622-4d789d084810 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:13.997115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:13.997737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:13.998012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:13.998276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.004350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:14.005518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.007007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:23:14.020741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:14.021913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:14.023128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 557/1098] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:14.027091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10bc1c-e242-4c7e-b635-0e092942854e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:14.029457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:14.062026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:14.062724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:14.062996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:14.063273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.063992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=5738d9ca67834a42b22fead73a050b99, user.id=06c40a9b2ec54b608991e3a8a90625b9) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:14.064293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.065700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.071395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 542/1099] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:13 2022] GET /identity/v3/role_assignments?user.id=06c40a9b2ec54b608991e3a8a90625b9&scope.project.id=5738d9ca67834a42b22fead73a050b99&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:14.131958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc02c75-8816-4801-afb2-662130239b54 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:14.151559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:14.176911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:14.177530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:14.177804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:14.178072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.178833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:14.179136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.180586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.201184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 558/1100] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Oct 4 08:23:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:14.207324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:14.324315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab35e7f-4c9a-45b2-b201-679a9b7c873f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:14.358466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:14.359091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:14.359366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:14.359640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.376308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:14.444832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.446200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.546448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 543/1101] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:14.552260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:14.663450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d375a6-89aa-42bd-9204-0e381dc816ca None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:14.700180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:14.700728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:14.700998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:14.701261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.712422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.714028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:14.730690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:14.731801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:23:14.732693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:14.732693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:23:14.732693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:14.734743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 559/1102] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:14.737530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:14.783763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:14.836470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fb9f19-a9fb-4d30-b1a9-2f9d9c227fcc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:14.869436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:14.870039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:14.870311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:14.870853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:14.871477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:14.871772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:14.873108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:14.883930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 544/1103] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:23:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:14.889610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:14.911992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:14.991874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8a41c1-0fdf-41f1-9e07-2c2580c4d3db None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:15.029231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:15.029872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:15.030149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:15.030419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:15.038168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:15.039687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:23:15.052597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:15.053844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:15.054828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 560/1104] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:23:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:15.059880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:15.174772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29664bc2-2e7a-488b-99ea-19931ffcd2f5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:15.184101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:15.218278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:15.218913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:15.219187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:15.219454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:15.220145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:15.220446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:15.221867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df18d2e1-4f02-4aba-b36b-6d25eb71dc2d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:15.235294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 545/1105] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:23:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:15.241944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:15.335287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:15.357492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bf1456-226b-4a90-8b0d-9f8333638a0e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:15.396381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:15.397205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:15.397322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:15.397534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:15.398281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] RBAC: Adding query filter params (role.id=951cca4e78364f3a8ea36c290df8e5f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=02d91fccc1c8400392fbbc34416f552b) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:15.398610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:15.400089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bd40db-77f7-4c77-9150-6704374747d8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:15.407351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 561/1106] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:15 2022] GET /identity/v3/role_assignments?user.id=02d91fccc1c8400392fbbc34416f552b&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=951cca4e78364f3a8ea36c290df8e5f3 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:16.290121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f293a02-d578-40e7-9769-00ff08ea1a9f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:16.291523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f293a02-d578-40e7-9769-00ff08ea1a9f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:16.291523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f293a02-d578-40e7-9769-00ff08ea1a9f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:16.292667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 546/1107] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:23:16.298486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:16.299008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:16.299100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:16.317629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:16.319048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:16.393097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 562/1108] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:16.666872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f293a02-d578-40e7-9769-00ff08ea1a9f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:16.773217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f293a02-d578-40e7-9769-00ff08ea1a9f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:16.810825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:16.811448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:16.811720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:16.811984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:16.813104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:16.814516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb23bed-b974-4aed-9761-1fc50ecb68fb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:16.816092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 547/1109] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:16.822353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:16.900363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:16.926451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce08ee7-98a2-4175-906e-b225a89954d7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:16.964906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:16.965549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:16.965828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:16.966097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:16.966794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:16.968242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f0c822-53b1-46fd-b00b-380aee0301ae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:16.990634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 563/1110] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:23:16 2022] POST /identity/v3/users => generated 286 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:19.263426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4664e8-a98b-4c02-84e8-2793918720f4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:19.263989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4664e8-a98b-4c02-84e8-2793918720f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:19.263989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4664e8-a98b-4c02-84e8-2793918720f4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:19.265307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 548/1111] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:19.271213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:19.271714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:19.271830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:19.291864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:19.293062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:19.366754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 564/1112] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:19.636391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4664e8-a98b-4c02-84e8-2793918720f4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:19.671846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:19.740339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4664e8-a98b-4c02-84e8-2793918720f4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:19.778264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:19.778915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:19.779185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:19.779446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] PATH_INFO: `/v3/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:19.780804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] RBAC: Authorizing `identity:get_role(role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:19.782160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:19.783498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 549/1113] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:19 2022] GET /identity/v3/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 223 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:19.789361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:19.848363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:19.892517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f69e6-8c20-450d-9582-1bc3404f82e0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:19.928768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:19.929401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:19.929679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:19.929955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] PATH_INFO: `/v3/users/bd640877be694f01bb9ae0fc4b8c8deb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:19.943364 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] RBAC: Authorizing `identity:get_user(user_id=bd640877be694f01bb9ae0fc4b8c8deb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:19.944821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:19.946469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 565/1114] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:19 2022] GET /identity/v3/users/bd640877be694f01bb9ae0fc4b8c8deb => generated 286 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:19.953941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:20.053254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00865bae-e9bd-42d2-b496-9518e6dd876c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:20.054805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:20.089591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:20.090192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:20.090467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:20.090763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:20.162743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:20.164189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:20.276016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 550/1115] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:20.282085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:20.393548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4fcb24-d105-4ea4-84d1-9392469eb1dc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:20.410845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:20.429397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:20.430020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:20.430291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:20.430569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:20.432260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:20.433692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787f7768-ea49-47e4-b034-1623429408d1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:20.435357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 566/1116] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:20 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:20.441593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:20.443792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:20.549358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c82d71c-9c5e-419b-8e80-34ab8a159599 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:20.583396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:20.586125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:20.586407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:20.586725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:20.587461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=5738d9ca67834a42b22fead73a050b99, user.id=bd640877be694f01bb9ae0fc4b8c8deb) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:20.587937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:20.589438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699dd177-7cca-42cd-9e44-ec0df5e83603 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:20.595026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 551/1117] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:20 2022] GET /identity/v3/role_assignments?user.id=bd640877be694f01bb9ae0fc4b8c8deb&scope.project.id=5738d9ca67834a42b22fead73a050b99&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:23:22.723419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dfadd-47aa-43ad-b47d-bb0ef64fd2bf None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:22.723948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dfadd-47aa-43ad-b47d-bb0ef64fd2bf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:22.723948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7dfadd-47aa-43ad-b47d-bb0ef64fd2bf None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:22.725283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 567/1118] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:22.730471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:22.730975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:22.731068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:22.749277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:22.750447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:22.809169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:22.816472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 552/1119] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:23.092923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7dfadd-47aa-43ad-b47d-bb0ef64fd2bf None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:23.196081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7dfadd-47aa-43ad-b47d-bb0ef64fd2bf None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:23.228830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:23.229442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:23.229712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:23.229975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] PATH_INFO: `/v3/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:23.231355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] RBAC: Authorizing `identity:get_role(role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:23.232707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:23.234009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 568/1120] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:23 2022] GET /identity/v3/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 223 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:23.239954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:23.334913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be618fe4-7738-4c6c-b566-3e298f5b189e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:23.366840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:23.367386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:23.367659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:23.367925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] PATH_INFO: `/v3/users/bd640877be694f01bb9ae0fc4b8c8deb` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:23.369536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] RBAC: Authorizing `identity:get_user(user_id=bd640877be694f01bb9ae0fc4b8c8deb)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:23.371311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:23.372837 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 553/1121] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:23 2022] GET /identity/v3/users/bd640877be694f01bb9ae0fc4b8c8deb => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:23.378952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:23.433085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:23.475373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c171f4-227a-456f-b8e7-f4c8dad50761 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:23.509101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:23.509744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:23.510023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:23.510297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:23.511434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:23.512868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abcf957-4d9c-45ee-9a3a-8290f77302c6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:23.514494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 569/1122] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:23 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:23.520458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:23.579987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:23.616287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415e57a-497a-45ec-95e3-19ad35c9a739 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:23.648836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:23.649474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:23.649744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:23.650007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99/users/bd640877be694f01bb9ae0fc4b8c8deb/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:23.652212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] RBAC: Authorizing `identity:create_grant(project_id=5738d9ca67834a42b22fead73a050b99, user_id=bd640877be694f01bb9ae0fc4b8c8deb, role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:23.653614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c602982-b2a3-4e65-95dc-abaa3586fa14 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:23.666436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 554/1123] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:23:23 2022] PUT /identity/v3/projects/5738d9ca67834a42b22fead73a050b99/users/bd640877be694f01bb9ae0fc4b8c8deb/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:23:25.885097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea44d24d-a490-49e2-a8b3-f86340428c3a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:25.885641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea44d24d-a490-49e2-a8b3-f86340428c3a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:25.885881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea44d24d-a490-49e2-a8b3-f86340428c3a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:25.887268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 570/1124] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:25.892828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:25.893280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:25.893373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:25.912017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:25.912701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:25.984502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 555/1125] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:26.253491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea44d24d-a490-49e2-a8b3-f86340428c3a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:26.360079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea44d24d-a490-49e2-a8b3-f86340428c3a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:26.383830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:26.396179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:26.396802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:26.397077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:26.397345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] PATH_INFO: `/v3/roles/97e78d101fab432cbfed8ce9b5fbd599` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:26.398757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] RBAC: Authorizing `identity:get_role(role_id=97e78d101fab432cbfed8ce9b5fbd599)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:26.400708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:26.402389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 571/1126] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:26 2022] GET /identity/v3/roles/97e78d101fab432cbfed8ce9b5fbd599 => generated 223 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:26.408339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:26.464586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:26.511760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2b475b-d575-42d0-9002-3f89bd7eee56 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:26.545571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:26.546260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:26.546535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:26.546834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] PATH_INFO: `/v3/users/bd640877be694f01bb9ae0fc4b8c8deb` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:26.547949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] RBAC: Authorizing `identity:get_user(user_id=bd640877be694f01bb9ae0fc4b8c8deb)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:26.549365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:26.551187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 556/1127] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:26 2022] GET /identity/v3/users/bd640877be694f01bb9ae0fc4b8c8deb => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:26.556936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:26.662055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff63a65-e962-4123-ac1b-264be8c5be10 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:26.695429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:26.696034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:26.696313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:26.696583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:26.756369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:26.767662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:26.769133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:26.867861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 572/1128] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:26.874698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:26.976111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb181cc-e13a-4dc5-b67f-84a12d7204ea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:27.000719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:27.014996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:27.015624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:27.015894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:27.016162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] PATH_INFO: `/v3/projects/5738d9ca67834a42b22fead73a050b99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:27.017296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] RBAC: Authorizing `identity:get_project(project_id=5738d9ca67834a42b22fead73a050b99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:27.018847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1cfb9e-30f0-4234-bf61-68c867589d3a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:27.020461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 557/1129] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:26 2022] GET /identity/v3/projects/5738d9ca67834a42b22fead73a050b99 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:27.027152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:27.119613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:27.129693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff21be-8145-4bee-9e7c-24ea34021b8e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:27.167599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:27.168199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:27.168479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:27.168749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:27.169467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] RBAC: Adding query filter params (role.id=97e78d101fab432cbfed8ce9b5fbd599, scope.project.id=5738d9ca67834a42b22fead73a050b99, user.id=bd640877be694f01bb9ae0fc4b8c8deb) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:27.169768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:27.171239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78acd55-cd2a-42e7-b7a7-e82f8aecffa1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:27.177180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 573/1130] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:27 2022] GET /identity/v3/role_assignments?user.id=bd640877be694f01bb9ae0fc4b8c8deb&scope.project.id=5738d9ca67834a42b22fead73a050b99&role.id=97e78d101fab432cbfed8ce9b5fbd599 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:29.354023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97fe326-c008-405d-b4bf-621a950e7585 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:29.354733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97fe326-c008-405d-b4bf-621a950e7585 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:29.354733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97fe326-c008-405d-b4bf-621a950e7585 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:29.355892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 558/1131] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:29.361284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:29.361734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:29.361828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:29.382096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:29.383316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:29.452868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 574/1132] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:29.719237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97fe326-c008-405d-b4bf-621a950e7585 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:29.824499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97fe326-c008-405d-b4bf-621a950e7585 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:29.859028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:29.859645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:29.859917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:29.860182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:29.861274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:29.862711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5772afd9-7462-46fa-ab96-8d51ef91af5c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:29.864267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 559/1133] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:29.871239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:29.959021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:29.975432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00932af8-d917-435a-831d-49f2f65f6c1a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:30.010732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:30.011356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:30.011633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:30.011905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:30.012576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:30.013988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d664224d-a464-4703-a49d-f6f4a09fc735 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:30.039756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 575/1134] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:23:29 2022] POST /identity/v3/projects => generated 311 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:32.341531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f911fe-5df0-4663-86a2-24942b873bba None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:32.342105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f911fe-5df0-4663-86a2-24942b873bba None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:32.342105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f911fe-5df0-4663-86a2-24942b873bba None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:32.343429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 560/1135] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:32.348632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:32.349020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:32.349179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:32.368614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:32.370139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:32.436147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 576/1136] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:32.708804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f911fe-5df0-4663-86a2-24942b873bba None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:32.770215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:32.815125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f911fe-5df0-4663-86a2-24942b873bba None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:32.849482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:32.850100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:32.850371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:32.850675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:32.851773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:32.853196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5a5b5a-a9d2-4408-8d88-817f542ea841 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:32.855355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 561/1137] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:23:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:32.861189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:32.932668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:32.970858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c968a-4574-4feb-b389-6028832c281a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:33.005805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:33.006408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:33.006710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:33.006979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:33.007638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:33.009068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6e934-c204-4602-831d-c5af884d1a06 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:33.036702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 577/1138] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:23:32 2022] POST /identity/v3/users => generated 286 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:35.254357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57f09b-9941-4559-849c-e527b03a9cbe None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:35.254954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57f09b-9941-4559-849c-e527b03a9cbe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:35.254954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57f09b-9941-4559-849c-e527b03a9cbe None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:35.256266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 562/1139] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:35.261797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:35.262670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:35.262670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:35.281622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:35.282840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:35.350539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 578/1140] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:35.622307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57f09b-9941-4559-849c-e527b03a9cbe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:35.725483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57f09b-9941-4559-849c-e527b03a9cbe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:35.751284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:35.763035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:35.763660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:35.763930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:35.764192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:35.774465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:35.779741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:35.780949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:35.782067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 563/1141] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:35.787007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:35.838773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:35.885196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d33c67-8705-4322-9aa8-362cdb11a537 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:35.918425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:35.919063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:35.919338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:35.919605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:35.920263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:35.920560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:35.921928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:35.930162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 579/1142] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:35.937318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:36.039198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7cc0a0-6106-4972-b829-cc5c98b49a10 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:36.075004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:36.075617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:36.075888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:36.076150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] PATH_INFO: `/v3/users/ce73af41a1f24eb9aaf14969045842b1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:36.089786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] RBAC: Authorizing `identity:get_user(user_id=ce73af41a1f24eb9aaf14969045842b1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:36.091249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:36.092871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 564/1143] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:35 2022] GET /identity/v3/users/ce73af41a1f24eb9aaf14969045842b1 => generated 286 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:36.098787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:36.206455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa24cc2-561c-4811-a88d-eb6df92e7d74 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:36.222592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:36.243426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:36.244168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:36.244839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:36.245113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:36.317956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:36.319847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:36.417010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 580/1144] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:36.422977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:36.469259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:36.524467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8259a2f-c272-45e3-a58c-6b6cf0c353ec None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:36.558982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:36.559607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:36.559883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:36.560151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] PATH_INFO: `/v3/projects/ef1c813d48404df3bf8cca00ba4bac84` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:36.561261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] RBAC: Authorizing `identity:get_project(project_id=ef1c813d48404df3bf8cca00ba4bac84)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:36.562714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba264b27-273e-4646-98b4-fe90f34875b0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:36.564349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 565/1145] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:36 2022] GET /identity/v3/projects/ef1c813d48404df3bf8cca00ba4bac84 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:36.570566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:36.586395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:36.676872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff5f7f7-02d6-4a7a-8d7f-0cbf7ac1b949 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:36.712024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:36.712666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:36.713477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:36.713754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:36.714475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=ef1c813d48404df3bf8cca00ba4bac84, user.id=ce73af41a1f24eb9aaf14969045842b1) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:36.714806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:36.716238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dffa36c-2218-4383-ac0c-9e038217f69e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:36.721734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 581/1146] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:36 2022] GET /identity/v3/role_assignments?user.id=ce73af41a1f24eb9aaf14969045842b1&scope.project.id=ef1c813d48404df3bf8cca00ba4bac84&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:38.859580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff14ae-b152-4d78-9db9-d4b201db0371 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:38.860142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff14ae-b152-4d78-9db9-d4b201db0371 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:38.860142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff14ae-b152-4d78-9db9-d4b201db0371 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:38.861488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 566/1147] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:38.866893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:38.867518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:38.867518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:38.886784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:38.888002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:38.957631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 582/1148] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:39.230856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff14ae-b152-4d78-9db9-d4b201db0371 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:39.328344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eff14ae-b152-4d78-9db9-d4b201db0371 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:39.341272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:39.362324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:39.363137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:39.363411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:39.363674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:39.374134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:39.378836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:39.379946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:39.380959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 567/1149] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:39.386010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:39.395864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:39.481555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f6defe-9c7e-4851-b816-2a61d2096aec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:39.514113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:39.514728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:39.515003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:39.515269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:39.515915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:39.516210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:39.517559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:39.525672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 583/1150] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:39.531519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:39.629426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bf6a38-0b77-4ddb-aad1-d5039cef6750 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:39.662534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:39.663172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:39.663447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:39.663711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] PATH_INFO: `/v3/users/ce73af41a1f24eb9aaf14969045842b1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:39.664812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] RBAC: Authorizing `identity:get_user(user_id=ce73af41a1f24eb9aaf14969045842b1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:39.667108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:39.668656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 568/1151] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:39 2022] GET /identity/v3/users/ce73af41a1f24eb9aaf14969045842b1 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:39.674424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:39.741635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:39.772922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69a91d-837b-4eb3-91c2-e11b444b2787 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:39.807037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:39.807661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:39.807939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:39.808214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] PATH_INFO: `/v3/projects/ef1c813d48404df3bf8cca00ba4bac84` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:39.809326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] RBAC: Authorizing `identity:get_project(project_id=ef1c813d48404df3bf8cca00ba4bac84)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:39.810779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36628151-979d-4c74-a5d8-055959b89e4d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:39.812396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 584/1152] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:39 2022] GET /identity/v3/projects/ef1c813d48404df3bf8cca00ba4bac84 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:39.818299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:39.860825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:39.917551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae89159-6ccc-4c09-9b9c-d6e734ce575c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:39.950104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:39.950769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:39.951040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:39.951307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] PATH_INFO: `/v3/projects/ef1c813d48404df3bf8cca00ba4bac84/users/ce73af41a1f24eb9aaf14969045842b1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:39.953156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] RBAC: Authorizing `identity:create_grant(project_id=ef1c813d48404df3bf8cca00ba4bac84, user_id=ce73af41a1f24eb9aaf14969045842b1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:39.954585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e158a4-8b85-4fc1-80e0-965b21f5a6fe None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:39.967648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 569/1153] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:23:39 2022] PUT /identity/v3/projects/ef1c813d48404df3bf8cca00ba4bac84/users/ce73af41a1f24eb9aaf14969045842b1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:23:42.154497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb754ed-98f2-4bb4-bcbd-cff9fd18099c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:42.154977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb754ed-98f2-4bb4-bcbd-cff9fd18099c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:42.155076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb754ed-98f2-4bb4-bcbd-cff9fd18099c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:42.156412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 585/1154] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:42.161683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:42.162105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:42.162201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:42.180660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:42.182791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:42.248591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 570/1155] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:42.531140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb754ed-98f2-4bb4-bcbd-cff9fd18099c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:42.630341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb754ed-98f2-4bb4-bcbd-cff9fd18099c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:42.662987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:42.663604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:42.663887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:42.664156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:42.670851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:42.675314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:42.680229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:42.681332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:42.683454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 586/1156] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:42.687957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:42.716898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:42.786632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fff914-9604-4456-9e09-25412c0fbe17 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:42.819335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:42.819921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:42.820191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:42.820455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:42.821104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:42.821398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:42.823338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:42.831084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 571/1157] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:42.837162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:42.937760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3babf8-ad7d-48ba-b83f-56458576e495 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:42.970683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:42.971312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:42.971591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:42.971864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] PATH_INFO: `/v3/users/ce73af41a1f24eb9aaf14969045842b1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:42.972981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] RBAC: Authorizing `identity:get_user(user_id=ce73af41a1f24eb9aaf14969045842b1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:42.974427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:42.976048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 587/1158] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:42 2022] GET /identity/v3/users/ce73af41a1f24eb9aaf14969045842b1 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:42.982703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:43.072494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:43.081103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4d8c9-f784-4649-a6cf-c4ef44de3270 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:43.113695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:43.114289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:43.114584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:43.114858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:43.182529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:43.183992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:43.272409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:43.282287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 572/1159] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:43.288344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:43.354415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:43.392469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e3dae-220a-4db4-a996-9e94e7ed63d3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:43.426193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:43.426841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:43.427119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:43.427393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] PATH_INFO: `/v3/projects/ef1c813d48404df3bf8cca00ba4bac84` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:43.428502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] RBAC: Authorizing `identity:get_project(project_id=ef1c813d48404df3bf8cca00ba4bac84)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:43.430056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4eac84-629a-45b7-83e9-b4026cf53592 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:43.431880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 588/1160] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:43 2022] GET /identity/v3/projects/ef1c813d48404df3bf8cca00ba4bac84 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:43.438204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:43.534198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5cb06-d6ad-42e4-832c-b5e3908f805f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:43.567280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:43.567841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:43.568118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:43.568385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:43.569095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=ef1c813d48404df3bf8cca00ba4bac84, user.id=ce73af41a1f24eb9aaf14969045842b1) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:43.569397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:43.570850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bd62c1-1559-4457-9866-f4ee18508a99 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:43.586769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 573/1161] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:43 2022] GET /identity/v3/role_assignments?user.id=ce73af41a1f24eb9aaf14969045842b1&scope.project.id=ef1c813d48404df3bf8cca00ba4bac84&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:45.762373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea9af4f-c46d-4d54-8424-f46b0685b953 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:45.762946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea9af4f-c46d-4d54-8424-f46b0685b953 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:45.762946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea9af4f-c46d-4d54-8424-f46b0685b953 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:45.764316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 589/1162] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:45.770144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:45.770602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:45.770727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:45.793733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:45.793733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:45.844945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:45.863289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 574/1163] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:46.141753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea9af4f-c46d-4d54-8424-f46b0685b953 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:46.251760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea9af4f-c46d-4d54-8424-f46b0685b953 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:46.286561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:46.287168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:46.287446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:46.287715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:46.290389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:46.369452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:46.371527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:46.474485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 590/1164] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:46.480678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:46.582771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bac259-bdeb-47bb-93de-ab1fd9528f1e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:46.616446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:46.617074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:46.617352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:46.617623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:46.625457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:46.626927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8e6edc1d-0f5e-4c9c-865a-617115f3fe8d None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:46.639810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:46.641113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:23:46.642335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 575/1165] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:23:46 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:46.647093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:46.680232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:46.750370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050dd1c7-a9b2-41d4-867e-7cec309e26f3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:46.784292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:46.784939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:46.785160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:46.785431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:46.786075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:46.786377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:46.787774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1e550c-ba45-4e87-8a69-84def7680033 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:46.796017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 591/1166] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Oct 4 08:23:46 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:48.957881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd322cd-c301-42e4-a377-eb23c0afc910 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:48.958442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd322cd-c301-42e4-a377-eb23c0afc910 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:48.958442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd322cd-c301-42e4-a377-eb23c0afc910 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:48.959756 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 576/1167] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:48.965351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe8344-f364-4055-bb2d-84c855aa710c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:48.965862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe8344-f364-4055-bb2d-84c855aa710c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:48.965954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe8344-f364-4055-bb2d-84c855aa710c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:48.986170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-71fe8344-f364-4055-bb2d-84c855aa710c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:48.987402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-71fe8344-f364-4055-bb2d-84c855aa710c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:49.059228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 592/1168] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:49.328852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd322cd-c301-42e4-a377-eb23c0afc910 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:49.377728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:49.432532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd322cd-c301-42e4-a377-eb23c0afc910 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:49.471422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:49.472025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:49.472295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:49.472556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:49.473217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:49.474598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bff90b-c0d7-468a-b78c-dbd3e9db00f9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:49.491850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 577/1169] 10.222.0.16 () {60 vars in 1138 bytes} [Tue Oct 4 08:23:49 2022] POST /identity/v3/domains => generated 267 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:51.697627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0092ea-6e83-4b95-96de-7cb5129d8d84 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:51.698163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0092ea-6e83-4b95-96de-7cb5129d8d84 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:51.698383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0092ea-6e83-4b95-96de-7cb5129d8d84 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:51.700366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 593/1170] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:51.705916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:51.706361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:51.706449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:51.724712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:51.725881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:51.793132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 578/1171] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:23:52.065833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0092ea-6e83-4b95-96de-7cb5129d8d84 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:52.093188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:52.168892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0092ea-6e83-4b95-96de-7cb5129d8d84 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:52.203818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:52.204446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:52.204715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:52.204983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] PATH_INFO: `/v3/domains/f1e86e074908446ab9db557bf24ec1c7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:52.215601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] RBAC: Authorizing `identity:get_domain(domain_id=f1e86e074908446ab9db557bf24ec1c7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:52.217161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b620f8-5fee-462f-93a3-4612fc418973 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:52.218832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 594/1172] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Oct 4 08:23:52 2022] GET /identity/v3/domains/f1e86e074908446ab9db557bf24ec1c7 => generated 267 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:52.224817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:52.260626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:52.334152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fcaa28-5967-47ac-a138-3028d8a47df9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:52.367552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:52.368160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:52.368436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:52.368704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:52.369912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:52.371391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265b7e66-b4fd-40b6-8323-bb38c150aa8c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:52.401407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 579/1173] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:23:52 2022] POST /identity/v3/projects => generated 361 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:54.622652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5086324-17fe-4894-9dbb-b0845c9594ce None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:54.623158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5086324-17fe-4894-9dbb-b0845c9594ce None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:54.623158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5086324-17fe-4894-9dbb-b0845c9594ce None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:54.624442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 595/1174] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:54.630088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:54.630606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:54.630700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:54.649139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:54.650896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:54.721235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 580/1175] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:55.004196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5086324-17fe-4894-9dbb-b0845c9594ce None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:55.104355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:55.117995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5086324-17fe-4894-9dbb-b0845c9594ce None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:55.154519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:55.155323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:55.155447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:55.155712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] PATH_INFO: `/v3/domains/f1e86e074908446ab9db557bf24ec1c7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:55.156865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=f1e86e074908446ab9db557bf24ec1c7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:55.158296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e00f750-b004-4d58-9a90-7590f5731b8d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:55.159926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 596/1176] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Oct 4 08:23:55 2022] GET /identity/v3/domains/f1e86e074908446ab9db557bf24ec1c7 => generated 267 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:55.165697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:55.185023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:55.266433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756131fb-1a88-4227-b84f-bd988b903301 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:55.301220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:55.301830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:55.302105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:55.302373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:55.303075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:55.304495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cee1a1e-7cbf-4958-a7bb-fc428f5d8a88 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:55.328172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 581/1177] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:23:55 2022] POST /identity/v3/users => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:23:57.639573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f355316-2dcf-478e-b79d-849626211c72 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:57.640127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f355316-2dcf-478e-b79d-849626211c72 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:57.640127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f355316-2dcf-478e-b79d-849626211c72 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:57.641439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 597/1178] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:23:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:23:57.646895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:57.647350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:57.647533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:57.665288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:23:57.666470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:23:57.734161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 582/1179] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:23:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 04 08:23:58.001778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f355316-2dcf-478e-b79d-849626211c72 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:58.110059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f355316-2dcf-478e-b79d-849626211c72 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:58.145507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:58.146145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:58.146426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:58.146736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:58.159007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:23:58.163792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:23:58.165059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:23:58.166621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 598/1180] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:23:58.171695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:58.179796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:58.286090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d9534a-b4e1-422d-979a-5d48c6613edd None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:58.320223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:58.320830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:58.321104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:58.321373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:58.322064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:58.322367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:58.323769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:58.332108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 583/1181] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:58.338685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:58.357424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:58.442850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7a9b16-8977-41b2-8e8a-eb743dce7c3b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:58.477759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:58.478430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:58.479237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:58.479503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] PATH_INFO: `/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:58.493236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] RBAC: Authorizing `identity:get_user(user_id=69c127a9f35b443b8f281ee9a2c6dcd1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:58.494988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:58.496638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 599/1182] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:58.502773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:58.579832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:58.615536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253c9676-b1e8-45ab-99b7-16507e6f45c4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:58.653782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:58.654456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:58.654656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:58.654918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:58.726399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:58.728047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:58.799556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:58.837204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 584/1183] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:23:58.843714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:58.920347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:23:58.945396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5f88d-b452-473c-888c-88a03a91efda None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:58.981143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:58.981800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:58.982078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:58.982352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] PATH_INFO: `/v3/projects/7cc9462cbf3c43b893c319e817a755f7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:58.983499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] RBAC: Authorizing `identity:get_project(project_id=7cc9462cbf3c43b893c319e817a755f7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:58.984926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2eda0b-b0e4-430d-b1dc-754ecc31bb99 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:58.986602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 600/1184] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/projects/7cc9462cbf3c43b893c319e817a755f7 => generated 361 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:23:58.992853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:23:59.099282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c88a6b-5638-49b9-a095-e946e946dc71 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:23:59.133459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:23:59.134061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:23:59.134338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:23:59.135240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:23:59.136147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=7cc9462cbf3c43b893c319e817a755f7, user.id=69c127a9f35b443b8f281ee9a2c6dcd1) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:23:59.136452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:23:59.137863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e308b86e-cadd-4168-8f46-28ebaf86b11a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:23:59.143632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 585/1185] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:23:58 2022] GET /identity/v3/role_assignments?user.id=69c127a9f35b443b8f281ee9a2c6dcd1&scope.project.id=7cc9462cbf3c43b893c319e817a755f7&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:01.285657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e924262-5977-410f-a9f4-128f9438b321 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:01.286178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e924262-5977-410f-a9f4-128f9438b321 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:01.286178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e924262-5977-410f-a9f4-128f9438b321 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:01.287555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 601/1186] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:24:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:24:01.293144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:01.293637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:01.293738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:01.315484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:24:01.316680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:24:01.344233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:01.390678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 586/1187] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:24:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:24:01.661462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e924262-5977-410f-a9f4-128f9438b321 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:01.765161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e924262-5977-410f-a9f4-128f9438b321 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:01.799802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:01.800421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:01.800692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:01.800955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:01.811803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:24:01.816547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:24:01.818260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:24:01.819795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 602/1188] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:24:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:24:01.824437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:01.924165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e06db1-f6c2-48df-8d9a-c04f102e8752 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:01.958009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:01.958655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:01.958935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:01.959205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:01.959873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:24:01.960173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:01.961548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:01.970815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 587/1189] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:24:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:01.978267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:01.981822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:02.083093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7dc91-36f6-4c19-9bfe-400d297e0440 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:02.116822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:02.117444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:02.117715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:02.117978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] PATH_INFO: `/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:02.119688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] RBAC: Authorizing `identity:get_user(user_id=69c127a9f35b443b8f281ee9a2c6dcd1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:02.121261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:02.123118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 603/1190] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:24:01 2022] GET /identity/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:24:02.128980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:02.155356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:02.233062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe43944-8c4a-4033-92e3-4630e9ba65f0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:02.267973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:02.268601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:02.268870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:02.269133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] PATH_INFO: `/v3/projects/7cc9462cbf3c43b893c319e817a755f7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:02.270902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] RBAC: Authorizing `identity:get_project(project_id=7cc9462cbf3c43b893c319e817a755f7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:02.271701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b11833-3f6c-408c-9db6-1f3122c0a802 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:02.273305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 588/1191] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:24:02 2022] GET /identity/v3/projects/7cc9462cbf3c43b893c319e817a755f7 => generated 361 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:02.279765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:02.350837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:02.388074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3ca8e-2b6f-41b6-80de-947b02dade34 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:02.422575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:02.423237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:02.423511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:02.423783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] PATH_INFO: `/v3/projects/7cc9462cbf3c43b893c319e817a755f7/users/69c127a9f35b443b8f281ee9a2c6dcd1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:02.425679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] RBAC: Authorizing `identity:create_grant(project_id=7cc9462cbf3c43b893c319e817a755f7, user_id=69c127a9f35b443b8f281ee9a2c6dcd1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:02.427156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26377d09-7cdb-4246-82d0-ddd4834a60e5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:02.440958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 604/1192] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:24:02 2022] PUT /identity/v3/projects/7cc9462cbf3c43b893c319e817a755f7/users/69c127a9f35b443b8f281ee9a2c6dcd1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:24:04.617597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c4e41-b585-4efe-b4b1-ce4aca659859 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:04.618200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c4e41-b585-4efe-b4b1-ce4aca659859 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:04.618200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89c4e41-b585-4efe-b4b1-ce4aca659859 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:04.619662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 589/1193] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:24:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:24:04.625636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:04.626129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:04.626242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:04.645308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:24:04.646507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:24:04.718789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 605/1194] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:24:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:24:04.992954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89c4e41-b585-4efe-b4b1-ce4aca659859 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:05.086320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:05.098259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89c4e41-b585-4efe-b4b1-ce4aca659859 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:05.141178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:05.141812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:05.142085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:05.142353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:05.153963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:24:05.158871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:24:05.160735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:24:05.162423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 590/1195] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:24:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:24:05.166665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:05.237713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:05.281050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23069612-b348-4893-b4f3-b882f2bc8aa7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:05.316177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:05.316784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:05.317061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:05.317333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:05.318001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:24:05.318304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:05.319705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:05.328176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 606/1196] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:24:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:05.334263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:05.446675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800e011b-5356-4f6a-babe-4b513a6c5d59 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:05.491674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:05.492306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:05.492581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:05.492848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] PATH_INFO: `/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:05.493966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] RBAC: Authorizing `identity:get_user(user_id=69c127a9f35b443b8f281ee9a2c6dcd1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:05.495429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:05.497018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 591/1197] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Oct 4 08:24:05 2022] GET /identity/v3/users/69c127a9f35b443b8f281ee9a2c6dcd1 => generated 311 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:05.503270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:05.617281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415d7ba-3bbc-470f-aa74-6a998990f74c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:05.641633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:05.654274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:05.654910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:05.655190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:05.655463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:05.723258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:05.724716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:05.826803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 607/1198] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:24:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:24:05.835200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:05.859587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:05.954518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b4113-fc8a-4f41-aa97-a3c1c6df7229 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:05.988250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:05.988876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:05.989140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:05.989407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] PATH_INFO: `/v3/projects/7cc9462cbf3c43b893c319e817a755f7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:05.990518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] RBAC: Authorizing `identity:get_project(project_id=7cc9462cbf3c43b893c319e817a755f7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:05.991969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88045780-ee4a-4e59-a37b-9c0d940ae5ac None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:05.999802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 592/1199] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Oct 4 08:24:05 2022] GET /identity/v3/projects/7cc9462cbf3c43b893c319e817a755f7 => generated 361 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:24:06.000986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:24:06.035291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:06.114163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ff99d-bfec-42d2-8c2d-a0f7dacaaff3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:24:06.151631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:24:06.152244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:24:06.152510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:24:06.152778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:24:06.153501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=7cc9462cbf3c43b893c319e817a755f7, user.id=69c127a9f35b443b8f281ee9a2c6dcd1) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:24:06.153800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:24:06.155271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f1d388-4e41-4687-98d2-951a10f17fb5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:24:06.161642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 608/1200] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:24:05 2022] GET /identity/v3/role_assignments?user.id=69c127a9f35b443b8f281ee9a2c6dcd1&scope.project.id=7cc9462cbf3c43b893c319e817a755f7&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:24:15.987516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=57 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:24:16.158594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=57 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:03.217749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0cf32-c48b-417e-8b80-974b8c04f3e5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:03.217749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0cf32-c48b-417e-8b80-974b8c04f3e5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:03.217749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0cf32-c48b-417e-8b80-974b8c04f3e5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:03.217749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 593/1201] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:03.222584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:03.226177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:03.226177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:03.245569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:03.246785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:03.350792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 609/1202] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:03.625830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0cf32-c48b-417e-8b80-974b8c04f3e5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:03.743104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0cf32-c48b-417e-8b80-974b8c04f3e5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:03.787774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:03.788414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:03.788788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:03.789090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:03.802091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:03.803616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:03.819344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:03.822815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:03.823856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:03.823856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:03.823856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:03.823856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 594/1203] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:25:03.826788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:03.957211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86891f5f-ded8-4c0b-a165-c01ad5ec44ec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:03.994227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:04.001310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:04.002386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:04.002814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:04.003029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:04.003682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:04.004071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:04.005575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812ffcf4-7fb6-471f-a4a1-12e902987397 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:04.029034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 610/1204] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:04.030338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:04.098729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:04.159864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ffeb98-de6d-4d37-9ad9-15de69bfbe2b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:04.201873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:04.202922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:04.203319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:04.203649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:04.204311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:04.205996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c8e5cd-97ec-45df-b95b-29f5c1b86584 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:04.234202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 595/1205] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Oct 4 08:25:04 2022] POST /identity/v3/users => generated 251 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:25:06.817592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd792a-1e54-48a7-9038-e9c7bef866b2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:06.818182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd792a-1e54-48a7-9038-e9c7bef866b2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:06.818182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd792a-1e54-48a7-9038-e9c7bef866b2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:06.819864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 611/1206] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:06.827040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:06.827608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:06.827720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:06.849473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:06.850924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:06.942594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 596/1207] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:07.228426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcd792a-1e54-48a7-9038-e9c7bef866b2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:07.365390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcd792a-1e54-48a7-9038-e9c7bef866b2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:07.407412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:07.408019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:07.408312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:07.408511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:07.420442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:07.425495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:07.426801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:07.427721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 612/1208] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:25:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:07.434378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:07.482063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:07.560954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9c7c9c-f6e5-4f5a-9f9c-9480a95e0f28 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:07.604903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:07.605516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:07.605788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:07.606050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:07.606741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:07.607035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:07.608385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:07.620587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 597/1209] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:25:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:07.626888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:07.697436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:07.763858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1ea84-8837-483b-a970-a78f67e8392c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:07.820977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:07.822281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:07.822736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:07.823067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:07.837382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:07.840113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:07.859331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:07.860415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:07.861298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:07.861298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:07.861298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:07.861298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 613/1210] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:07.866686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:07.982758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:07.987199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03122212-e6b0-40f2-b554-c6efb7d4d2af None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:08.027810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:08.028695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:08.028827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:08.029088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:08.029746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:08.030064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:08.031739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:08.048966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 598/1211] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:25:08.052263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:08.177628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d543d2d-ab90-496d-a625-fccaf25bd85b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:08.179584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:08.225179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:08.226206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:08.226612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:08.227002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:08.236418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:08.238460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:08.252682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:08.253654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:08.254699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:08.255892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 614/1212] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:08.261081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:08.379071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da78872-9d8b-4b8f-9aa8-10f9fdd9a3d6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:08.421536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:08.422177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:08.422478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:08.422826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:08.423746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:08.424203 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:08.425665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:08.450581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 599/1213] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:25:08.452824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:08.575385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8154c29f-ef13-404f-a965-9427e1432fee None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:08.621049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:08.621996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:08.622451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:08.622949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:08.657228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:08.700210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:08.702444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:08.835568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 615/1214] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:25:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:25:08.837241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:08.861197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:08.956558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc47fd-7cb2-4876-a3c9-3399a3775ab4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:08.999234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:09.000338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:09.000804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:09.001285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:09.014977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:09.017043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:09.033693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:09.034731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:09.035628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:09.035628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:09.035628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:09.037415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 600/1215] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:09.043036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:09.107988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:09.167891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6946-78b8-4ce0-8930-35598bc5b658 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:09.208551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:09.209217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:09.209474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:09.209786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:09.210504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:09.210814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:09.212222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:09.230410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 616/1216] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:09.231703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:09.326311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:09.364020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f32a9cd-38e9-4771-b6c9-7a9f4131ddd9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:09.410338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:09.411085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:09.411371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:09.411579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:09.422086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:09.423713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:09.439689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:09.441429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:09.442343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 601/1217] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:09.447886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:09.590303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd63613-4a4d-4ff4-bbf9-a75bc6e1e6f2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:09.617023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:09.630694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:09.631457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:09.631736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:09.632119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:09.632946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:09.633267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:09.635655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e67c6-2f90-458a-b592-5c77407a68eb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:09.650441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 617/1218] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:09.658407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:09.772841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51824b56-b112-4084-b816-8987b235ab6f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:09.809172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:09.817116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:09.817950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:09.818148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:09.818430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:09.819327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=18e6e8451d334251a87f84b03e57a4c8) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:09.819742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:09.821255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58f24c-b41b-4b9d-b2c8-d6963291f02d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:09.828265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 602/1219] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:25:09 2022] GET /identity/v3/role_assignments?user.id=18e6e8451d334251a87f84b03e57a4c8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:12.270202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252f6a1-ed8a-4e44-9fc3-74f48c25abcc None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:12.270752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252f6a1-ed8a-4e44-9fc3-74f48c25abcc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:12.270880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252f6a1-ed8a-4e44-9fc3-74f48c25abcc None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:12.272138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 618/1220] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:25:12.277633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:12.278123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:12.278219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:12.302340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:12.302340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:12.383718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 603/1221] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:12.682069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252f6a1-ed8a-4e44-9fc3-74f48c25abcc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:12.819092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252f6a1-ed8a-4e44-9fc3-74f48c25abcc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:12.861230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:12.862179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:12.862461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:12.862770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:12.879195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:12.887632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:12.888966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:12.890204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 619/1222] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:25:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:25:12.895312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:13.016248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604d48b-56a1-4a12-bb05-1668a71b41d8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:13.043606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:13.054565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:13.055188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:13.055464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:13.055736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:13.056574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:13.056926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:13.058325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:13.070791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 604/1223] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:25:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:13.073211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:13.119100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:13.186801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eceb2d5a-4a36-49f6-a61f-c7ee610031df None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:13.229494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:13.230117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:13.230392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:13.230694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:13.241758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:13.243630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:13.262730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:13.264131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:13.265382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:13.265382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:13.265382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:13.265382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 620/1224] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:13.271185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:13.390233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a5ffae-27bd-4d9b-a45b-319662255151 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:13.434025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:13.434736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:13.435029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:13.435306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:13.435952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:13.436241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:13.437690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:13.451278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 605/1225] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:13.458942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:13.590129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:13.599632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aee8a9-c6d7-453a-a2f3-90e4f1afd1d7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:13.640040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:13.640688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:13.641005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:13.641274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:13.649004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:13.650579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:13.670949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:13.672402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:13.673984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 621/1226] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:13.679164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:13.745895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:13.814211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591c25bf-d4f4-4d08-a7c1-a910e5d097d8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:13.861017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:13.861609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:13.861884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:13.863447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:13.863447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:13.863447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:13.864692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:13.892790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 606/1227] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:13.898867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:14.008141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ec4387-352f-458f-938d-941b5d48f04c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:14.050099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:14.051161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:14.051656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:14.052120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:14.071199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:14.073958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:14.089425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:14.090812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:14.092328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:14.092328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:14.092328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:14.095792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 622/1228] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:14.096876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:14.230517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b5e7dc-29d5-4c1a-a44a-000860ada080 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:14.232846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:14.270513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:14.271260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:14.271567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:14.271843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:14.272697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:14.273143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:14.274931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:14.293718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 607/1229] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:25:14.295443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:14.306596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:14.414293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cf2d4-97ba-4ca8-9ebf-019d5a3d406b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:14.452387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:14.453037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:14.453339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:14.453551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:14.460765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:14.462242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:14.474711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:14.475862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:14.477026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 623/1230] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:14.487325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:14.605620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e09aa9-0798-4a19-a40e-c91668e5eb4d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:14.648330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:14.649366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:14.650030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:14.650632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:14.651672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:14.652195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:14.654185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d5ad1c-6aa5-4410-8757-ad8076e4d8fd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:14.672810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 608/1231] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:14.681182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:14.755292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:14.833360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef5f55b-481a-4b85-b6cc-d5f490aa8d62 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:14.872516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:14.873173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:14.873449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:14.873717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/18e6e8451d334251a87f84b03e57a4c8/roles/e07e16b2134c42f0aa93fa8b9a0776f3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:14.889553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=18e6e8451d334251a87f84b03e57a4c8, role_id=e07e16b2134c42f0aa93fa8b9a0776f3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:14.891038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c66b821-94a7-4a8d-be0f-c1020be4ff50 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:14.904446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 624/1232] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:25:14 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/18e6e8451d334251a87f84b03e57a4c8/roles/e07e16b2134c42f0aa93fa8b9a0776f3 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:25:17.194860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21577708-24ea-4c9c-81f0-6c10f14853d4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:17.195738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21577708-24ea-4c9c-81f0-6c10f14853d4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:17.195738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21577708-24ea-4c9c-81f0-6c10f14853d4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:17.197574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 609/1233] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:17.203944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:17.204721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:17.204721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:17.225222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:17.226603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:17.302611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 625/1234] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:17.612332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21577708-24ea-4c9c-81f0-6c10f14853d4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:17.643807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:17.751642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21577708-24ea-4c9c-81f0-6c10f14853d4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:17.794887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:17.795832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:17.796091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:17.796361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:17.808402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:17.813231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:17.814332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:17.815537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 610/1235] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Oct 4 08:25:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:17.821932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:17.878769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:17.939566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e2977f-7847-47d9-91a9-4763d5a4c1c4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:17.976576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:17.977261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:17.977542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:17.977873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:17.979103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:17.979513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:17.980902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:17.989491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 626/1236] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Oct 4 08:25:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:17.998185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:18.101848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:18.116955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1678925-5425-4695-ad8f-a3b7860755f8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:18.157657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:18.158262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:18.158562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:18.158844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:18.169150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:18.170602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:18.184972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:18.186136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:18.188534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:18.188534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:18.188534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:18.188534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 611/1237] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:18.193459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:18.315782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d12cabe-b2c9-4f35-b506-1c4250b2e123 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:18.328004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:18.360213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:18.360950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:18.361308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:18.361665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:18.362473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:18.362908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:18.364495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:18.382422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 627/1238] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:18.390074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:18.503548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db0e1cc-8484-486a-91ea-20cda5c84f01 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:18.541659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:18.542276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:18.542582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:18.542842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:18.545898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:18.553417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:18.555048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:18.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:18.578702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:18.580175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 612/1239] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:18 2022] GET /identity/v3/users/placement => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:18.583105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:18.709494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9304d-f928-4a5b-8597-b1cb55b53f08 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:18.748350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:18.749214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:18.749493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:18.749758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:18.750487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:18.750816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:18.752253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:18.771750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 628/1240] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:18.778740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:18.906957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce508c2-7e1a-48dc-af4a-e5b566be8fea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:18.955773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:18.956431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:18.956694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:18.956996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:19.027140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:19.034138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:19.035625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:19.155639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 613/1241] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:25:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:25:19.157545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:19.187147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:19.281028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376c99c-3c34-4bde-b910-5fb8fec3db4a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:19.320418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:19.321159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:19.321476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:19.322350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:19.334235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:19.335740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:19.356923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:19.358047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:19.358047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:19.358047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:19.358047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 629/1242] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:19.363344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:19.495198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:19.502677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24f68ca-9862-462a-9004-29b7bd69d5b3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:19.546832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:19.547411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:19.547792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:19.548091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:19.548745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:19.549045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:19.551357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:19.567083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 614/1243] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:19.573746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:19.674237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:19.715335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495af428-1db0-476b-a639-91436d4beb6d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:19.765233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:19.766121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:19.766413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:19.766887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:19.778821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:19.781274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:19.799682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:19.801323 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:19.804497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 630/1244] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:25:19.807920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:19.943528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853d89b-4a63-4a99-b2cd-58e554039a35 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:19.985833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:19.986436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:19.986739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:19.987049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:19.987927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:19.988132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:19.990491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c627cc-a994-4c9c-bdf5-a612f98dbae8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:20.005110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 615/1245] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:20.015398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:20.179408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0652f2a-2ae1-433d-9f0e-2505e635fb93 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:20.226158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:20.227056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:20.227391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:20.227665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:20.229677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] RBAC: Adding query filter params (role.id=e07e16b2134c42f0aa93fa8b9a0776f3, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=18e6e8451d334251a87f84b03e57a4c8) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:20.230180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:20.233395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c69de3-cda7-427e-9dff-96f300b217d7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:20.242952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 631/1246] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:25:20 2022] GET /identity/v3/role_assignments?user.id=18e6e8451d334251a87f84b03e57a4c8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=e07e16b2134c42f0aa93fa8b9a0776f3 => generated 608 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:25:22.677253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c35d28-dd17-4b17-89e5-bbf0b8477476 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:22.677979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c35d28-dd17-4b17-89e5-bbf0b8477476 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:22.677979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c35d28-dd17-4b17-89e5-bbf0b8477476 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:22.679168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 616/1247] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:25:22.685336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:22.685832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:22.685921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:22.701911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:22.704630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:22.706430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:22.788045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 632/1248] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:23.073591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c35d28-dd17-4b17-89e5-bbf0b8477476 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:23.093431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:23.194653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c35d28-dd17-4b17-89e5-bbf0b8477476 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:23.229856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:23.230485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:23.230788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:23.231054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:23.241574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:23.246723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:23.247872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:23.248754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 617/1249] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:25:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:23.254232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:23.361960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90f6abd-5f38-4c43-94f3-bc3e4e16ab26 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:23.393888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:23.401124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:23.401743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:23.402021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:23.402293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:23.403028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:23.403301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:23.404850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:23.413973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 633/1250] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:25:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:23.420331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:23.507208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:23.533624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375336bf-036e-4b36-9a57-2a16c06bdadf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:23.580918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:23.582076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:23.582499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:23.582987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:23.597178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:23.598964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:23.620074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:23.621482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:23.622375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:23.622375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:23.622375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:23.622375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 618/1251] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:23.627475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:23.738397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbfe0fd-a53f-485a-a888-c036db0b1b16 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:23.777463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:23.778140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:23.778459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:23.778723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:23.779422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:23.779732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:23.781148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:23.796575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 634/1252] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:23.803292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:23.938975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83103c6-0082-4a3c-866d-386c535a8a0a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:23.967988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:23.976537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:23.977183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:23.977461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:23.977730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:23.986432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:23.987950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:24.001138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:24.002367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:24.003571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 619/1253] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:24.008457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:24.066499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:24.113020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a543bda9-a44d-4f7f-b086-b8e97004bb3e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:24.152614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:24.153239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:24.153511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:24.153776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:24.154569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:24.154874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:24.156337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:24.176617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 635/1254] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:24.183659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:24.305985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb965c9f-7793-41bc-9265-7cd5cb2e22f4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:24.345526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:24.346139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:24.346413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:24.346721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:24.404654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:24.418210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:24.420148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:24.518991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 620/1255] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:25:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:25:24.526265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:24.622697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26ba82c-30b5-4cca-826e-90752dccb35e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:24.635395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:24.655989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:24.656602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:24.656874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:24.657136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:24.667304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:24.668744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:24.682836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:24.683906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:24.685082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:24.685082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:24.685082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:24.685082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 636/1256] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:24.689845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:24.776071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:24.792107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d96c4-6c08-4176-a12c-b84a5b00f79b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:24.829936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:24.830612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:24.830826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:24.831093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:24.831728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:24.832025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:24.833366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:24.845158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 621/1257] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:24.850849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:24.952694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f66d65-0896-4cb1-b99b-510785cb5c55 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:24.985442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:24.986059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:24.986338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:24.986643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:24.993078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:24.995115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:25.006291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:25.007366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:25.008612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 637/1258] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:25.013307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:25.110954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e722e6-26c3-4961-a2f8-dad01494f2e0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:25.146063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:25.146709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:25.146987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:25.147248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:25.147932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:25.148227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:25.149635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d85f3-334e-489e-8df9-b9c866606d23 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:25.161695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 622/1259] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:25.168259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:25.193448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:25.267713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6c95d3-ea65-4f02-b046-d7a569ccf9e1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:25.300070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:25.300661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:25.300934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:25.301200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:25.301907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=18e6e8451d334251a87f84b03e57a4c8) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:25.302206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:25.303658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca21cab-b0dd-4742-86c6-c0b77f3ef069 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:25.309003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 638/1260] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:25:25 2022] GET /identity/v3/role_assignments?user.id=18e6e8451d334251a87f84b03e57a4c8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:27.441526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c9f35-3e0b-43d0-8adc-129ccef2696e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:27.442026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c9f35-3e0b-43d0-8adc-129ccef2696e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:27.442026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c9f35-3e0b-43d0-8adc-129ccef2696e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:27.444534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 623/1261] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:27.451064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:27.451064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:27.451064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:27.469550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:27.470766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:27.538192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 639/1262] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:27.825538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53c9f35-3e0b-43d0-8adc-129ccef2696e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:27.838454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:27.945301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53c9f35-3e0b-43d0-8adc-129ccef2696e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:27.977580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:27.978741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:27.979030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:27.979304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:27.988987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:27.994528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:27.995648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:27.996567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 624/1263] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:25:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:28.002673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:28.022573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:28.117883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff162e6-97c8-42cc-84d9-b1d28d0b10ce None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:28.157942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:28.158582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:28.158852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:28.159119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:28.159777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:28.160081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:28.161441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:28.170354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 640/1264] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:28.176108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:28.251274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:28.293905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c31848-e4dd-49a6-a06f-22e4c69caf5d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:28.327642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:28.328262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:28.328536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:28.328801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:28.339214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:28.340641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:28.355273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:28.356395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:28.357468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:28.357468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:28.357468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:28.357468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 625/1265] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:28.362214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:28.446776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:28.467918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60e42f2-37de-45b6-af94-30f4fcb88046 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:28.500960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:28.501679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:28.502017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:28.502352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:28.503185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:28.503570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:28.505210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:28.518499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 641/1266] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:28.524360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:28.639922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64694440-5b05-4d29-914f-3a21e3e3c0bf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:28.665107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:28.682697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:28.683371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:28.683721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:28.684068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:28.693352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:28.695184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:28.709730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:28.710843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:28.718044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 626/1267] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:28.718957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:28.829668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb0fa2-be54-47f6-8c53-aebbe692f453 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:28.863836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:28.864429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:28.864705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:28.864974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:28.865688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:28.865988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:28.867473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:28.877474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:28.888187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 642/1268] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:28.894039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:29.000444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462c86b-5565-46f0-8bb7-791d66f85719 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:29.037723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:29.038342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:29.038637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:29.038940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:29.050495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:29.051999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:29.067324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:29.068503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:29.069883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:29.069883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:29.069883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:29.069883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 627/1269] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:25:29.075641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:29.201040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608d2b5-5032-4aa6-ad98-f2f3dfc461a0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:29.234516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:29.235142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:29.235412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:29.235678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:29.236310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:29.236604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:29.237961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:29.250194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 643/1270] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:29.255962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:29.291144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:29.361430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280780a9-cfc5-4ecd-a578-83a05531176b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:29.396134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:29.396753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:29.397030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:29.397300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:29.405162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:29.406625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:29.418767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:29.420962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:29.422563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 628/1271] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:29.426483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:29.486796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:29.531802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f6f9d7-9f1e-4142-9bd5-78560dd4a119 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:29.567501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:29.568113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:29.568385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:29.568655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:29.569344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:29.569644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:29.571108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e660f6d4-fc51-4a5a-81d6-f405e352ebe7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:29.584164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 644/1272] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:29.591476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:29.687624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:29.697739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed72aa-672d-4632-8449-34343b873769 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:29.731536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:29.732260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:29.732468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:29.732740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] PATH_INFO: `/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/18e6e8451d334251a87f84b03e57a4c8/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:29.734718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] RBAC: Authorizing `identity:create_grant(project_id=427c33aa73c94f0e9e383089fb5f6fb1, user_id=18e6e8451d334251a87f84b03e57a4c8, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:29.736290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22365fdb-ac64-41a2-a1bc-74f88695238c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:29.752197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 629/1273] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Oct 4 08:25:29 2022] PUT /identity/v3/projects/427c33aa73c94f0e9e383089fb5f6fb1/users/18e6e8451d334251a87f84b03e57a4c8/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:25:31.943339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0329cbf2-f160-41d0-94c8-b4d67cb9bce5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:31.943941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0329cbf2-f160-41d0-94c8-b4d67cb9bce5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:31.943941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0329cbf2-f160-41d0-94c8-b4d67cb9bce5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:31.945222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 645/1274] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:31.951073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:31.951620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:31.951620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:31.969737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:31.970961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:32.039072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 630/1275] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:32.308933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0329cbf2-f160-41d0-94c8-b4d67cb9bce5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:32.412043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0329cbf2-f160-41d0-94c8-b4d67cb9bce5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:32.432537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:32.450659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:32.451292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:32.451575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:32.451842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:32.463269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 04 08:25:32.468245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:32.469347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:32.470611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 646/1276] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Oct 4 08:25:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:32.475466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:32.555796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:32.575899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b0c24-6958-41dc-b009-db9546cbe1d2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:32.609403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:32.610003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:32.610277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:32.610579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:32.611225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:32.611523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:32.612868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:32.621154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 631/1277] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Oct 4 08:25:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:32.627322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:32.730877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048a125d-49b2-42fb-bcc0-5e68e9093a22 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:32.766428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:32.767197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:32.767519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:32.767834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:32.781326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:32.783088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:32.801875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:32.809350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:32.810790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:32.810790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:32.810790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:32.810790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 647/1278] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:32.811128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:32.921365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6903711d-b63b-408c-8c57-3c3e3a1ebbbe None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:32.958960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:32.959602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:32.959848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:32.960120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:32.960770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:32.961072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:32.962700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:32.975035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 632/1279] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:32.980767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:33.012673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:33.082855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e2a663-2132-4789-a75d-5d95cdc27a3f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:33.117320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:33.117950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:33.118229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:33.118587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:33.126752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:33.128230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 04 08:25:33.140776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:33.141963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:33.142977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 648/1280] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:33.147934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:33.152527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:33.256425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1fde3e-cd0c-4ba7-9635-52425af9980c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:33.293480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:33.294251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:33.294533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:33.294971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:33.295887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:33.296224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:33.297982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:33.319044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 633/1281] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Oct 4 08:25:33 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:33.325532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:33.425092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:33.435115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad71fd-b3b4-43bf-b0e7-7b251387a4c2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:33.473192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:33.473790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:33.474065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:33.474328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:33.550353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:33.551754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:33.648484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:33.659633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 649/1282] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Oct 4 08:25:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 04 08:25:33.665809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:33.726574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:33.766172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33676e1-4604-46b1-ab4e-343cc66e57c8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:33.801171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:33.801733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:33.802007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:33.802275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:33.813410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:33.814889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 04 08:25:33.830116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:33.831319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 04 08:25:33.832431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 04 08:25:33.832431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 04 08:25:33.832431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:33.832431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 634/1283] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:25:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 04 08:25:33.837680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:33.941025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21877d0-1257-4de3-81d8-72af0a838f7c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:33.975743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:33.976335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:33.976606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:33.976873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:33.977511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:33.978348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:33.979758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:33.991682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 650/1284] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Oct 4 08:25:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:25:33.998074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:34.109146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80780964-54fe-433c-adcf-f7062de5b5e6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:34.123646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:34.144432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:34.145071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:34.145358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:34.145612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:34.153426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:34.154874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 04 08:25:34.167575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:34.168682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Oct 04 08:25:34.169979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 635/1285] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Oct 4 08:25:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:34.174619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:34.224359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:34.275844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bcced8-40fe-4540-8666-e42d337050d1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:34.310958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:34.311549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:34.311826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:34.312089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:34.318498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:34.318498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:34.318498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c49faf9-f90a-4ae4-ac77-c309c5ede951 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:34.328870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 651/1286] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Oct 4 08:25:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:34.335108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:34.444245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e04a9e-f3c4-42ce-b665-6f9f7ba00650 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:34.482493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:34.483145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:34.483423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:34.483687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:34.484426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] RBAC: Adding query filter params (role.id=d7f1381a769e433aa1b033022931be96, scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1, user.id=18e6e8451d334251a87f84b03e57a4c8) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:34.484730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:34.486167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2141e-46eb-4cfc-8ff9-868c599cc622 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:34.492607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 636/1287] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Oct 4 08:25:34 2022] GET /identity/v3/role_assignments?user.id=18e6e8451d334251a87f84b03e57a4c8&scope.project.id=427c33aa73c94f0e9e383089fb5f6fb1&role.id=d7f1381a769e433aa1b033022931be96 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:36.709595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75768bc6-c4a9-4903-b6bc-133a9721352d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:36.710116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75768bc6-c4a9-4903-b6bc-133a9721352d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:36.710116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75768bc6-c4a9-4903-b6bc-133a9721352d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:36.711488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 652/1288] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:36.717204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61975889-ecd0-49b9-be69-72801c016947 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:36.717713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61975889-ecd0-49b9-be69-72801c016947 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:36.717807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61975889-ecd0-49b9-be69-72801c016947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:36.735673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-61975889-ecd0-49b9-be69-72801c016947 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:36.736875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-61975889-ecd0-49b9-be69-72801c016947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:36.768559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:36.813171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 637/1289] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:37.086580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75768bc6-c4a9-4903-b6bc-133a9721352d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:37.197432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75768bc6-c4a9-4903-b6bc-133a9721352d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:37.208279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:37.232538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:37.233164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:37.233437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:37.233703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:37.234281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:37.235668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:37.244790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:37.246010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:37.246965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 653/1290] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:25:37 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:37.252256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61975889-ecd0-49b9-be69-72801c016947 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:37.352116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61975889-ecd0-49b9-be69-72801c016947 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:37.387928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:37.388530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:37.388803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:37.389072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:37.389695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:37.389999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:37.391383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5b079a-5fb5-4d91-b4fe-d6f3051bd0b4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:37.396066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 638/1291] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Oct 4 08:25:37 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:37.402044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:37.503842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062657a3-708f-4e01-838c-90daa8bc6036 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:37.536589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:37.537184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:37.537456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:37.537721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:37.538341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] RBAC: Adding query filter params (type=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:37.538661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:37.540020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0d474-4b4b-4b40-91b0-bccae188bd6a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:37.544373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 654/1292] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Oct 4 08:25:37 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:39.684506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7501ea68-71b0-40e6-be18-efa86dd877e4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:39.685068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7501ea68-71b0-40e6-be18-efa86dd877e4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:39.685068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7501ea68-71b0-40e6-be18-efa86dd877e4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:39.686345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 639/1293] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:39.691731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075fbf-d8b8-4ed5-b416-6cf1a63fd2ca None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:39.692126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075fbf-d8b8-4ed5-b416-6cf1a63fd2ca None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:39.692278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0075fbf-d8b8-4ed5-b416-6cf1a63fd2ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:39.710281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f0075fbf-d8b8-4ed5-b416-6cf1a63fd2ca None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:39.711515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f0075fbf-d8b8-4ed5-b416-6cf1a63fd2ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:39.741032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:39.780071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 655/1294] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:40.052640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7501ea68-71b0-40e6-be18-efa86dd877e4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:40.083751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:40.154890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7501ea68-71b0-40e6-be18-efa86dd877e4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:40.187133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:40.187729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:40.187999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:40.188262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:40.188920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:40.190261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d322e02-3be4-454f-ba42-e18987d236b0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:40.205153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 640/1295] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Oct 4 08:25:40 2022] POST /identity/v3/services => generated 246 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:25:42.430071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1cd2-b8e2-4b2e-a6be-4bbe48a4ec19 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:42.430653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1cd2-b8e2-4b2e-a6be-4bbe48a4ec19 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:42.430653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1cd2-b8e2-4b2e-a6be-4bbe48a4ec19 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:42.432709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 656/1296] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:42.439754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:42.440300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:42.440300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:42.459115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:42.460333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:42.534973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 641/1297] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:42 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:42.812038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b1cd2-b8e2-4b2e-a6be-4bbe48a4ec19 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:42.922056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b1cd2-b8e2-4b2e-a6be-4bbe48a4ec19 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:42.956357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:42.957144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:42.957255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:42.957525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:42.958120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:42.959524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:42.969785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:42.971093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:42.972126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 657/1298] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:25:42 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:42.977034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:42.995916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:43.076478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54be91-428c-492f-a393-a22ebe555099 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:43.109715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:43.110314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:43.110608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:43.110878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:43.111498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:43.111790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:43.113135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:43.118090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 642/1299] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Oct 4 08:25:42 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:43.124164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:43.156804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:43.235142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46970e8b-60c5-4d1a-88d3-b0bd4ebef688 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:43.274100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:43.275383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:43.275655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:43.275923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:43.276622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] RBAC: Adding query filter params (interface=public, service_id=9781527be7c942f39bf808bba556e72d, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:43.276923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:43.278289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d9f9df-c7e4-4350-b666-335f6da5ecec None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:43.283671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 658/1300] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Oct 4 08:25:43 2022] GET /identity/v3/endpoints?service_id=9781527be7c942f39bf808bba556e72d&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:43.289708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:43.378802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:43.401227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbe8a25-b8d0-4236-b1b8-236e8bdcb053 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:43.435827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:43.436424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:43.436712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:43.436963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:43.437562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:43.438949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be673e76-2146-4e12-a86f-132a10092c30 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:43.445498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 643/1301] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Oct 4 08:25:43 2022] GET /identity/v3/services => generated 2191 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:25:45.598512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2441c-096f-4d9f-bbee-dbd69b961dc0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:45.599074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2441c-096f-4d9f-bbee-dbd69b961dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:45.599074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2441c-096f-4d9f-bbee-dbd69b961dc0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:45.600333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 659/1302] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:25:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:25:45.606201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:45.606673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:45.606764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:45.625238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:25:45.626411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:25:45.693271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 644/1303] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:25:45 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 04 08:25:45.963033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2441c-096f-4d9f-bbee-dbd69b961dc0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:46.073062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:46.079876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2441c-096f-4d9f-bbee-dbd69b961dc0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:46.112962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:46.113596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:46.113878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:46.114148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:46.114769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:46.116166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:25:46.125601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:25:46.126789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:25:46.128037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 660/1304] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Oct 4 08:25:45 2022] GET /identity/v3/services/placement => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 04 08:25:46.132803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:46.221904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:46.244542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b792b7d3-9ea5-47e5-b013-df47ec71f59e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:46.278174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:46.278797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:46.279075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:46.279346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:46.279971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:25:46.280268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:46.281616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4315a94a-33e5-436f-b53a-9375b00f686a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:46.286738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 645/1305] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Oct 4 08:25:46 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:25:46.293037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:25:46.390854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af36c06-9e8f-45f1-85b8-f08006be6308 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:25:46.426967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:25:46.427578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:25:46.427854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:25:46.428124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:25:46.428808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:25:46.430185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b625092-0ae3-4731-96a5-abdddc60bc4a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:25:46.451123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 661/1306] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:25:46 2022] POST /identity/v3/endpoints => generated 331 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:25:56.283642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=30 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:25:56.446673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:52.218633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e701452-bb9c-4e8c-bf16-a881a4abee5c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:52.219632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e701452-bb9c-4e8c-bf16-a881a4abee5c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:52.220143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e701452-bb9c-4e8c-bf16-a881a4abee5c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:52.221829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 646/1307] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:26:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:26:52.227563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471dcd8-d23f-4dd2-a97a-39813770f098 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:52.228082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471dcd8-d23f-4dd2-a97a-39813770f098 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:52.228170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471dcd8-d23f-4dd2-a97a-39813770f098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:52.361969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9471dcd8-d23f-4dd2-a97a-39813770f098 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:26:52.363223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9471dcd8-d23f-4dd2-a97a-39813770f098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:52.491277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 662/1308] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:26:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:26:52.784449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:52.785106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:52.785482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:52.787783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 647/1309] 10.222.0.16 () {54 vars in 964 bytes} [Tue Oct 4 08:26:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:26:52.795804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9654e808-551e-4d1f-a80c-38dd2746e9f6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:52.796328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9654e808-551e-4d1f-a80c-38dd2746e9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:52.796785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9654e808-551e-4d1f-a80c-38dd2746e9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:52.881575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9654e808-551e-4d1f-a80c-38dd2746e9f6 None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:26:52.883381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9654e808-551e-4d1f-a80c-38dd2746e9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:52.907906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:52.982595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 663/1310] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Oct 4 08:26:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:26:52.993104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:26:52.995479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:53.144140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5dc186-1b9a-4c38-8aa7-fa5594311e73 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:26:53.239603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:26:53.240280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:53.240485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:53.240753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:53.242852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:53.312045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:53.377704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:26:53.379326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926355eb-1377-4ffa-816a-73d1870c1a9a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:26:53.401828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 648/1311] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:26:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:26:55.938542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:55.939006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:55.939098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:55.940439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 664/1312] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:26:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:26:55.946518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec958418-02f2-42d3-8593-7aadb41437de None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:55.947010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec958418-02f2-42d3-8593-7aadb41437de None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:55.947090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec958418-02f2-42d3-8593-7aadb41437de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:56.029940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ec958418-02f2-42d3-8593-7aadb41437de None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:26:56.031181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ec958418-02f2-42d3-8593-7aadb41437de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:56.127402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 649/1313] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:26:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:26:56.410045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:26:56.411601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:56.516395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93f095-ac43-4381-b5e3-3547cbbcc8f7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:26:56.588365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:56.590905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:26:56.591521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:56.591979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:56.592291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:56.594223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:56.701812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:26:56.703208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf54b7-5dfc-492e-8c9c-e54526936c91 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:26:56.721257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 665/1314] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:26:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:26:59.259160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:59.259690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:59.259811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:59.261071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 650/1315] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:26:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:26:59.266281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b179349-0469-4edc-966f-a5bc3ccfe0d8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:59.266690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b179349-0469-4edc-966f-a5bc3ccfe0d8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:59.266861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b179349-0469-4edc-966f-a5bc3ccfe0d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:59.331818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5b179349-0469-4edc-966f-a5bc3ccfe0d8 None None] MFA Rules not processed for user `829a99ce94f043fb93bddd51c7961fbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:26:59.333029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5b179349-0469-4edc-966f-a5bc3ccfe0d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:59.385553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:59.429265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 666/1316] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:26:59 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:26:59.717301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:26:59.718872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:59.728964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:26:59.822148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93d1325-db27-4760-bd43-6c5d383ea824 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:26:59.894168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:26:59.894791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:26:59.895062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:26:59.895332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:26:59.896899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:26:59.959148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:00.005456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:00.006818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c66cbd-600d-4230-90f6-d149f3e3f73b service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:00.025002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 651/1317] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:26:59 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:02.769585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:02.770133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:02.770133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:02.771485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 667/1318] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:02.778049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680b9bb-7089-45e3-8a57-be264867ff74 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:02.778518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680b9bb-7089-45e3-8a57-be264867ff74 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:02.778646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7680b9bb-7089-45e3-8a57-be264867ff74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:02.846504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7680b9bb-7089-45e3-8a57-be264867ff74 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:02.847760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7680b9bb-7089-45e3-8a57-be264867ff74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:02.956098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 652/1319] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:02 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:03.261188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:03.262751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:03.367674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6303af-7321-4cbc-86a0-186292a12670 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:03.445011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:03.446346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:03.446765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:03.447040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:03.448639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:03.455538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:03.564104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:03.565549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b67ce9-7439-47dd-aaba-642e5860efaa service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:03.583684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 668/1320] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:03 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:06.450175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:06.450733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:06.450733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:06.452287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 653/1321] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:06.458178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5920798-fac1-4cd3-9d67-1ac7f5fc2d5a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:06.458773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5920798-fac1-4cd3-9d67-1ac7f5fc2d5a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:06.458773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5920798-fac1-4cd3-9d67-1ac7f5fc2d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:06.535130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a5920798-fac1-4cd3-9d67-1ac7f5fc2d5a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:06.536377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a5920798-fac1-4cd3-9d67-1ac7f5fc2d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:06.603765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:06.647289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 669/1322] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:06.941649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:06.943239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:06.967902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:07.069358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfa755-214b-4244-b46f-e85b407962d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:07.156159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:07.156772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:07.157051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:07.157321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:07.159135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:07.232989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:07.272865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:07.274466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c17533c-4761-42d9-b995-56ac8afecc0a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:07.292545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 654/1323] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:09.601035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:09.602113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:09.602113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:09.603441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 670/1324] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:09.609362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba56d9-70b4-41a7-81fa-0989f0027c0d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:09.610037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba56d9-70b4-41a7-81fa-0989f0027c0d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:09.610258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba56d9-70b4-41a7-81fa-0989f0027c0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:09.699459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7aba56d9-70b4-41a7-81fa-0989f0027c0d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:09.700685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7aba56d9-70b4-41a7-81fa-0989f0027c0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:09.801100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 655/1325] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:10.083124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:10.084656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:10.211524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdecb63-8f64-45f5-a13d-00fd86f5ae64 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:10.292158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:10.292767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:10.293043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:10.293312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:10.294983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:10.315007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:10.415618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:10.417062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69c347-ec4e-4656-8381-ec0877e8dcf2 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:10.435768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 671/1326] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:12.724453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:12.724941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:12.725038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:12.726900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 656/1327] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:12.732585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08faab42-90dc-48ce-b1f7-c192025dc73e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:12.733073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08faab42-90dc-48ce-b1f7-c192025dc73e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:12.733190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08faab42-90dc-48ce-b1f7-c192025dc73e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:12.802201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-08faab42-90dc-48ce-b1f7-c192025dc73e None None] MFA Rules not processed for user `829a99ce94f043fb93bddd51c7961fbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:12.803433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-08faab42-90dc-48ce-b1f7-c192025dc73e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:12.868669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:12.901272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 672/1328] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:12 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:13.196194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:13.197751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:13.231093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:13.309896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ec0de-2c75-48cf-b26a-0ac12789dfe3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:13.398878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:13.399612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:13.399884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:13.400167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:13.401860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:13.487271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:13.521446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:13.522937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d880c2-9fe2-41f9-9fde-b1a81e17d3f1 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:13.542683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 657/1329] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:13 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:15.987528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:15.988076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:15.988076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:15.989434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 673/1330] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:15.995791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6db54-4d71-48ed-907d-cfd25bcc6afa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:15.996246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6db54-4d71-48ed-907d-cfd25bcc6afa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:15.996358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6db54-4d71-48ed-907d-cfd25bcc6afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:16.065182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-31d6db54-4d71-48ed-907d-cfd25bcc6afa None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:16.066400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-31d6db54-4d71-48ed-907d-cfd25bcc6afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:16.170263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 658/1331] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:16.450826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:16.452363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:16.562062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d38224-f509-4ab8-8088-e337bcdfa01d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:16.634973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:16.635569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:16.635838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:16.636099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:16.637682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:16.664722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:16.747477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:16.748831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baef39fa-4a2a-4c4e-8020-0533f96c2c61 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:16.767189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 674/1332] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:18.997975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:18.998509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:18.998509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:18.999852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 659/1333] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:19.005296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c91ca-43e6-4cd7-9224-fa34fc2cd92e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:19.005693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c91ca-43e6-4cd7-9224-fa34fc2cd92e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:19.005839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71c91ca-43e6-4cd7-9224-fa34fc2cd92e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:19.078816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e71c91ca-43e6-4cd7-9224-fa34fc2cd92e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:19.080020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e71c91ca-43e6-4cd7-9224-fa34fc2cd92e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:19.156093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:19.178933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 675/1334] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:19.467384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:19.468915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:19.509821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:19.576839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8637df9b-bace-403e-b2e2-e394f2135900 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:19.652657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:19.653266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:19.653539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:19.653806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:19.655429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:19.743939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:19.770823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:19.772194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee024567-df72-482d-bb04-d44629b5b4f9 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:19.791112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 660/1335] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:22.409800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:22.410339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:22.410460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:22.411738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 676/1336] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:22.417758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21f267-1b39-494e-8a27-0a210606fa59 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:22.418236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21f267-1b39-494e-8a27-0a210606fa59 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:22.418331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21f267-1b39-494e-8a27-0a210606fa59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:22.485155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ab21f267-1b39-494e-8a27-0a210606fa59 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:22.486375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ab21f267-1b39-494e-8a27-0a210606fa59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:22.581926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 661/1337] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:22 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:22.867019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:22.868615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:22.969002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2e74fa-8c20-4f52-8dd6-73d3c7711a7f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:23.042312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:23.042852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:23.043125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:23.043388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:23.044956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:23.080683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:23.156261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:23.157708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e0ffb8-d5ff-4f30-bd20-94b1f65cbc0d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:23.176150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 677/1338] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:22 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:27.317592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc68d5-617e-4787-8fbb-478f0c3a89c9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:27.318120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc68d5-617e-4787-8fbb-478f0c3a89c9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:27.318120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc68d5-617e-4787-8fbb-478f0c3a89c9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:27.319508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 662/1339] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:27.325000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155ed07d-da71-4d01-a84e-b186ed981519 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:27.325483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155ed07d-da71-4d01-a84e-b186ed981519 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:27.325668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155ed07d-da71-4d01-a84e-b186ed981519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:27.392793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-155ed07d-da71-4d01-a84e-b186ed981519 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:27.393933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-155ed07d-da71-4d01-a84e-b186ed981519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:27.483934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:27.496001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 678/1340] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:27.798138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:27.798424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:27.798731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:27.800074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 663/1341] 10.222.0.16 () {54 vars in 964 bytes} [Tue Oct 4 08:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:27.806911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df3122-7972-4966-ad76-292516b8b5e7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:27.807192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df3122-7972-4966-ad76-292516b8b5e7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:27.807460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df3122-7972-4966-ad76-292516b8b5e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:27.884405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-31df3122-7972-4966-ad76-292516b8b5e7 None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:27.885628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-31df3122-7972-4966-ad76-292516b8b5e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:27.962393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 679/1342] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Oct 4 08:27:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:27.972040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:27.973608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:28.036611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:28.094345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf366565-fa1b-4c7c-a809-075171d6a671 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:28.175506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:28.176117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:28.176390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:28.176656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:28.178274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:28.275520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:28.296148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:28.297534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9748e0-db57-4593-a451-1f5b37d9ce96 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:28.317844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 664/1343] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:31.751104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:31.751890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:31.751890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:31.753281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 680/1344] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:31.760090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53ff1f-acb8-47a8-9c79-0657b41413e1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:31.760638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53ff1f-acb8-47a8-9c79-0657b41413e1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:31.760638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a53ff1f-acb8-47a8-9c79-0657b41413e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:31.850327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4a53ff1f-acb8-47a8-9c79-0657b41413e1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:31.851718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4a53ff1f-acb8-47a8-9c79-0657b41413e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:31.952442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 665/1345] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:32.256268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:32.257933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:32.352314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:32.371924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0030fbdb-3ac3-4132-ba51-2dd693ed037f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:32.446489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:32.447126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:32.447404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:32.447673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:32.449281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:32.558346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:32.559749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb809a7-28c5-4481-bd4d-2a6210b6898c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:32.577582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 681/1346] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:32 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:27:37.048274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:37.048818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:37.048901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:37.050400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 666/1347] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:37.056759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96518c8-d7a8-40de-8a7a-1b970dcdb66a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:37.057248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96518c8-d7a8-40de-8a7a-1b970dcdb66a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:37.057453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96518c8-d7a8-40de-8a7a-1b970dcdb66a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:37.075068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:37.129224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f96518c8-d7a8-40de-8a7a-1b970dcdb66a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:37.130493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f96518c8-d7a8-40de-8a7a-1b970dcdb66a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:37.233407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 682/1348] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:27:37.522099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:37.523821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:37.584399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:37.631599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93733a4e-71ea-4297-9cdd-446e49bc5ea6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:37.705581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:37.706101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:37.706378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:37.706740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:37.708335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:37.818559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:37.823440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:37.824869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c88662-c993-48fa-b5b2-b0f1c908e49f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:37.843032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 667/1349] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:42.827842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3637f7-ee78-444c-ab9a-47e97207156e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:42.827842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3637f7-ee78-444c-ab9a-47e97207156e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:42.827842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3637f7-ee78-444c-ab9a-47e97207156e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:42.831828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 683/1350] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:27:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:42.835257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081259a-8a96-4df3-b275-0c368b4e83be None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:42.835878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081259a-8a96-4df3-b275-0c368b4e83be None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:42.836954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081259a-8a96-4df3-b275-0c368b4e83be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:42.950095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f081259a-8a96-4df3-b275-0c368b4e83be None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:42.951875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f081259a-8a96-4df3-b275-0c368b4e83be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:43.156483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 668/1351] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:27:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:27:47.231809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=59 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:47.453243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecd1cc-deb3-4b56-adaf-ddaece4265c7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:47.453584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecd1cc-deb3-4b56-adaf-ddaece4265c7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:47.453823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecd1cc-deb3-4b56-adaf-ddaece4265c7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:47.455553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 684/1352] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:27:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:47.460939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022a43-a16e-4ed7-a512-0e00c33891bb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:47.461764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022a43-a16e-4ed7-a512-0e00c33891bb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:47.462318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022a43-a16e-4ed7-a512-0e00c33891bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:47.520992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:47.549680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f9022a43-a16e-4ed7-a512-0e00c33891bb None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:47.551192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f9022a43-a16e-4ed7-a512-0e00c33891bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:47.675582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 669/1353] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:27:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:49.328346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41884de6-ca42-4f6c-9572-f5384483fb91 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:49.328899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41884de6-ca42-4f6c-9572-f5384483fb91 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:49.328899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41884de6-ca42-4f6c-9572-f5384483fb91 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:49.330236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 685/1354] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:27:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:49.335564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbab2c-a734-46c7-aa64-02a52bfe536b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:49.336606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbab2c-a734-46c7-aa64-02a52bfe536b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:49.336892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbab2c-a734-46c7-aa64-02a52bfe536b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:49.421623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1edbab2c-a734-46c7-aa64-02a52bfe536b None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:49.422783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1edbab2c-a734-46c7-aa64-02a52bfe536b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:49.471048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:49.540580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 670/1355] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:27:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:53.057602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208fbbb-959c-49fb-aa89-56aa2af62ccf None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:53.058128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208fbbb-959c-49fb-aa89-56aa2af62ccf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:53.058128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b208fbbb-959c-49fb-aa89-56aa2af62ccf None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:53.059495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 686/1356] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:53.065089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd40b8-ff72-4401-aacf-09a7b93265c8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:53.065518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd40b8-ff72-4401-aacf-09a7b93265c8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:53.065655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd40b8-ff72-4401-aacf-09a7b93265c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:53.144041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ddbd40b8-ff72-4401-aacf-09a7b93265c8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:53.145535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ddbd40b8-ff72-4401-aacf-09a7b93265c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:53.264277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 671/1357] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:53.571371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:53.572677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:53.573391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:53.575984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 687/1358] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:27:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:53.583439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b94ab-49dc-4c3a-8948-0bfb43252597 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:53.583794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b94ab-49dc-4c3a-8948-0bfb43252597 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:53.584120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b94ab-49dc-4c3a-8948-0bfb43252597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:53.606066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:53.673478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f48b94ab-49dc-4c3a-8948-0bfb43252597 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:53.674748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f48b94ab-49dc-4c3a-8948-0bfb43252597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:53.809370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 672/1359] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:27:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:27:53.817211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:53.818887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:53.981868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437eb9a8-e6ba-4770-a0fc-acea060aca84 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:54.063076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:54.114590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:54.115210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:54.115490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:54.115764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:54.118139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:54.221283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:54.222925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a953bc-2b74-4f06-ad12-ef857ed91e10 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:54.243384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 688/1360] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:27:56.693635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:56.694338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:56.694338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:56.702040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 673/1361] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:27:56 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:27:56.702730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0785-b983-4762-b8fe-3b674d75ff5d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:56.703204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0785-b983-4762-b8fe-3b674d75ff5d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:56.703204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e0785-b983-4762-b8fe-3b674d75ff5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:56.771082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3f4e0785-b983-4762-b8fe-3b674d75ff5d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:27:56.773315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3f4e0785-b983-4762-b8fe-3b674d75ff5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:56.815986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:56.880987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 689/1362] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:27:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:27:57.175252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:27:57.176840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:57.255494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:57.363876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9339fa30-fa8a-4fde-83fb-31adeb17d66a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:27:57.499544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:27:57.500155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:27:57.500432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:27:57.500701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:27:57.502604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:27:57.527339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:27:57.598772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:27:57.600331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf09dda-3944-45ec-a690-6fd06d2eab57 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:27:57.619541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 674/1363] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:27:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:00.152699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8078bc-5139-4bf5-ba25-c13577923306 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:00.153577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8078bc-5139-4bf5-ba25-c13577923306 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:00.154870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8078bc-5139-4bf5-ba25-c13577923306 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:00.157242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 690/1364] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:00 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:00.162416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512f53c-30fc-4c21-8b36-7c70b4e83a39 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:00.162914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512f53c-30fc-4c21-8b36-7c70b4e83a39 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:00.163164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6512f53c-30fc-4c21-8b36-7c70b4e83a39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:00.246766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6512f53c-30fc-4c21-8b36-7c70b4e83a39 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:00.247974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6512f53c-30fc-4c21-8b36-7c70b4e83a39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:00.389393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:00.398956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 675/1365] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:00.697901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:00.698731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:00.699417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:00.702577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 691/1366] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:28:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:00.708114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff807be-346b-4651-a7eb-6a8ead2ea59a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:00.708400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff807be-346b-4651-a7eb-6a8ead2ea59a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:00.708674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff807be-346b-4651-a7eb-6a8ead2ea59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:00.804922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bff807be-346b-4651-a7eb-6a8ead2ea59a None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:00.806139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bff807be-346b-4651-a7eb-6a8ead2ea59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:00.950724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 676/1367] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:28:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:00.955585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:00.958150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:01.156463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a35926c-c9be-4690-88ee-a13469325894 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:01.185247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:01.300402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:01.300933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:01.301207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:01.301490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:01.303295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:01.420436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:01.420778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022ff247-6883-4de8-8207-7d2fbdf48fc8 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:01.430536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633c1b6a-228e-4e04-8df1-6394d6172829 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:01.431150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633c1b6a-228e-4e04-8df1-6394d6172829 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:01.431150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633c1b6a-228e-4e04-8df1-6394d6172829 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:01.432622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 677/1368] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:28:01.442855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 692/1369] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 488 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:01.455000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2d063-b9a0-456b-b2cd-e7e1964f95a8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:01.455292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2d063-b9a0-456b-b2cd-e7e1964f95a8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:01.455561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2d063-b9a0-456b-b2cd-e7e1964f95a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:01.538621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:01.550327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e2d2d063-b9a0-456b-b2cd-e7e1964f95a8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:01.551349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e2d2d063-b9a0-456b-b2cd-e7e1964f95a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:01.702893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 678/1370] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:28:02.053377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:02.054295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:02.054807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:02.056726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 693/1371] 10.222.0.16 () {54 vars in 963 bytes} [Tue Oct 4 08:28:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:02.062389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab06333-2b25-4ead-9d27-77c55f582fe9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:02.062706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab06333-2b25-4ead-9d27-77c55f582fe9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:02.063027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab06333-2b25-4ead-9d27-77c55f582fe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:02.173737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5ab06333-2b25-4ead-9d27-77c55f582fe9 None None] MFA Rules not processed for user `546c67695c1e447da68b91637aefe135`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:02.175675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5ab06333-2b25-4ead-9d27-77c55f582fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:02.262515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:02.271999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 679/1372] 10.222.0.16 () {58 vars in 1060 bytes} [Tue Oct 4 08:28:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:02.279627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:02.281129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:02.313031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:02.418908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47ac92-44a8-4912-b009-736b3a742877 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:02.525637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:02.525637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:02.525637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:02.526175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:02.527592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:02.627361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:02.664874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:02.666173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eca37e-fabb-48ef-a69c-79f578022bd0 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:02.687179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 694/1373] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:03.801278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:03.801826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:03.801826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:03.803219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 680/1374] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:03.810453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e1aff-7375-4baa-b643-048dc45d707e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:03.810453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e1aff-7375-4baa-b643-048dc45d707e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:03.812055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e1aff-7375-4baa-b643-048dc45d707e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:03.885442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9e7e1aff-7375-4baa-b643-048dc45d707e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:03.886829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9e7e1aff-7375-4baa-b643-048dc45d707e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:03.993783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 695/1375] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:04.306100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:04.307840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:04.496081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5b6145-4ae3-4a4b-89e0-d74308146cbd None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:04.574989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:04.625751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:04.626370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:04.626929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:04.627197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:04.628782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:04.732089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:04.732888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64ead0d-6dae-4e30-8a38-b8025fd606b8 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:04.754023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 681/1376] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:07.059556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:07.060155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:07.060155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:07.061649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 696/1377] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:07.066847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f5b9de-d9e9-4d75-8222-909fe414e6f3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:07.067318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f5b9de-d9e9-4d75-8222-909fe414e6f3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:07.067421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f5b9de-d9e9-4d75-8222-909fe414e6f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:07.137581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-10f5b9de-d9e9-4d75-8222-909fe414e6f3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:07.138851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-10f5b9de-d9e9-4d75-8222-909fe414e6f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:07.182791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:07.271193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 682/1378] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:07.567058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:07.568710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:07.611810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:07.734144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a846cf43-ff3c-4cf6-898a-21a2d5881373 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:07.863517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:07.870390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:07.871202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:07.871486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:07.871753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:07.873327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:07.981877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:07.983818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22030c78-6c7b-4a6f-874d-829dbf6c97af service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:08.006662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 697/1379] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:10.259784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:10.260245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:10.260333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:10.261793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 683/1380] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:10.267057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed37a3a-07ef-44b5-a917-be60c9567e2c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:10.267470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed37a3a-07ef-44b5-a917-be60c9567e2c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:10.267674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed37a3a-07ef-44b5-a917-be60c9567e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:10.337823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1ed37a3a-07ef-44b5-a917-be60c9567e2c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:10.338998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1ed37a3a-07ef-44b5-a917-be60c9567e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:10.412255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:10.451836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 698/1381] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:10.746897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:10.748438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:10.918649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87309391-2cc5-4eda-ba54-41217da163c1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:10.932106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:11.055292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c436d881-3a26-407f-ab57-b166782806da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:11.055817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c436d881-3a26-407f-ab57-b166782806da service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:11.056094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c436d881-3a26-407f-ab57-b166782806da service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:11.056364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c436d881-3a26-407f-ab57-b166782806da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:11.057988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c436d881-3a26-407f-ab57-b166782806da service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:11.161221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c436d881-3a26-407f-ab57-b166782806da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:11.163568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c436d881-3a26-407f-ab57-b166782806da service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:11.183557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 684/1382] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:13.440550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:13.441189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:13.441189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:13.442367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 699/1383] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:13.448107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a1ebd-3c87-4804-bb4d-7b7c77a4890d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:13.448693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a1ebd-3c87-4804-bb4d-7b7c77a4890d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:13.448693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a1ebd-3c87-4804-bb4d-7b7c77a4890d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:13.470640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:13.531979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-192a1ebd-3c87-4804-bb4d-7b7c77a4890d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:13.534608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-192a1ebd-3c87-4804-bb4d-7b7c77a4890d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:13.647212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 685/1384] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:13.943944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:13.945561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:14.114395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1cf2a6-7246-4e30-885c-a16d93b544df None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:14.172543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:14.251067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:14.251941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:14.252215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:14.252482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:14.254098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:14.358526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:14.360183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda3fef7-cc87-436f-b5ff-9428afce409a service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:14.379645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 700/1385] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:16.608864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:16.609397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:16.609397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:16.610753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 686/1386] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:16.616710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93100ac4-ed06-4250-8deb-6ea66ea52488 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:16.617173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93100ac4-ed06-4250-8deb-6ea66ea52488 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:16.617271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93100ac4-ed06-4250-8deb-6ea66ea52488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:16.685929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-93100ac4-ed06-4250-8deb-6ea66ea52488 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:16.687161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-93100ac4-ed06-4250-8deb-6ea66ea52488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:16.695511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:16.785423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 701/1387] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:17.074659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:17.077107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:17.173928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:17.232747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e484c5-10aa-441a-a08c-beb7d895fe41 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:17.355159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:17.355770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:17.356040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:17.356302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:17.357900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:17.423750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:17.458308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:17.460247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048fc4a-36fb-407f-a893-1797dd59eaa8 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:17.479140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 687/1388] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:18.570797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4dda-b15b-4690-bacc-ff9db2cc3938 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:18.571332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4dda-b15b-4690-bacc-ff9db2cc3938 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:18.571332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4dda-b15b-4690-bacc-ff9db2cc3938 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:18.572711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 702/1389] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:18.578076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0088a-0adb-4aee-bf7f-f3f831d006de None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:18.578523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0088a-0adb-4aee-bf7f-f3f831d006de None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:18.578649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0088a-0adb-4aee-bf7f-f3f831d006de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:18.646228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f5d0088a-0adb-4aee-bf7f-f3f831d006de None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:18.647662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f5d0088a-0adb-4aee-bf7f-f3f831d006de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:18.764345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 688/1390] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:19.067488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:19.067850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:19.068218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:19.069627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 703/1391] 10.222.0.16 () {54 vars in 962 bytes} [Tue Oct 4 08:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:19.075151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eabf3ec-8ab6-49c4-8b43-cea5c62201d6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:19.075442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eabf3ec-8ab6-49c4-8b43-cea5c62201d6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:19.075718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eabf3ec-8ab6-49c4-8b43-cea5c62201d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:19.122855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:19.157745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3eabf3ec-8ab6-49c4-8b43-cea5c62201d6 None None] MFA Rules not processed for user `bbc7631562d645dd9b7876a59de53a31`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:19.159129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3eabf3ec-8ab6-49c4-8b43-cea5c62201d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:19.240009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 689/1392] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Oct 4 08:28:19 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:19.248030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:19.249572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:19.361635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49930d7c-c5e6-4792-a366-6f889b56cc98 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:19.403029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:19.440392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:19.441062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:19.441340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:19.441613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:19.443777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:19.556262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:19.557699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673de48f-9588-4df6-956d-64fc1acf335c service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:19.576663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 704/1393] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:19.723662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:19.724225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:19.724225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:19.725576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 690/1394] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:19.730993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ab129-c8b2-44f8-92b8-e55efefd8da2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:19.731279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ab129-c8b2-44f8-92b8-e55efefd8da2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:19.731553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ab129-c8b2-44f8-92b8-e55efefd8da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:19.807774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1c0ab129-c8b2-44f8-92b8-e55efefd8da2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:19.809083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1c0ab129-c8b2-44f8-92b8-e55efefd8da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:19.831670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:19.921254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 705/1395] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:20.228080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:20.229663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:20.354713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d53b9e-28b9-43c8-952e-89312f33381e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:20.355085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d53b9e-28b9-43c8-952e-89312f33381e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:20.355563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d53b9e-28b9-43c8-952e-89312f33381e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:20.383307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:20.402370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4c3905-bebd-4b52-8f80-a749d5988186 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:20.427060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-98d53b9e-28b9-43c8-952e-89312f33381e None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:20.429184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-98d53b9e-28b9-43c8-952e-89312f33381e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:20.517057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 706/1396] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:28:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:28:20.533354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:20.533969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:20.534240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:20.534503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:20.536241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:20.562770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58adca-a01d-4668-97eb-55c9d50a3aac None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:20.563059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58adca-a01d-4668-97eb-55c9d50a3aac None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:20.563335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58adca-a01d-4668-97eb-55c9d50a3aac None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:20.564725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 707/1397] 10.222.0.16 () {54 vars in 943 bytes} [Tue Oct 4 08:28:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:28:20.572502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baf6f7-959c-4ffd-9664-3821d51ffcd2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:20.572921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baf6f7-959c-4ffd-9664-3821d51ffcd2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:20.573190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baf6f7-959c-4ffd-9664-3821d51ffcd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:20.589889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:20.651212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:20.652792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:20.656748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:20.672505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 691/1398] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:20.687302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d1baf6f7-959c-4ffd-9664-3821d51ffcd2 None None] MFA Rules not processed for user `5cc208be4d444563929a03634cfddee7`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:20.688595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d1baf6f7-959c-4ffd-9664-3821d51ffcd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:20.813044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 708/1399] 10.222.0.16 () {58 vars in 1040 bytes} [Tue Oct 4 08:28:20 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:28:20.823649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:20.824925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:20.921883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadc4e9f-3177-44a4-8522-6c25f80cf13f service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:21.036504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:21.037894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:21.037894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:21.037894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:21.041646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:21.066714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:21.126452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:21.128119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece21694-258b-407e-84bd-6a873259e342 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:21.137367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 692/1400] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:28:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:28:22.995178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:22.995753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:22.995850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:22.998017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 709/1401] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:23.008216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2a0ef-d0de-4c8c-a5c0-dda7a0fdf2a4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:23.008216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2a0ef-d0de-4c8c-a5c0-dda7a0fdf2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:23.008216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2a0ef-d0de-4c8c-a5c0-dda7a0fdf2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:23.096100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-efa2a0ef-d0de-4c8c-a5c0-dda7a0fdf2a4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:23.098199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-efa2a0ef-d0de-4c8c-a5c0-dda7a0fdf2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:23.228246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 693/1402] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:23.549123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:23.550650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:23.632738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:23.718287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce9ad4b-f222-42e7-aaf5-9dcef97a0892 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:23.863532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a643967-8831-418f-854d-4fb295036916 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:23.864201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a643967-8831-418f-854d-4fb295036916 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:23.864482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a643967-8831-418f-854d-4fb295036916 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:23.864752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a643967-8831-418f-854d-4fb295036916 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:23.867185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a643967-8831-418f-854d-4fb295036916 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:23.908581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:23.978781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a643967-8831-418f-854d-4fb295036916 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:23.980651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a643967-8831-418f-854d-4fb295036916 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:24.002222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 710/1403] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:26.282838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:26.283376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:26.283376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:26.284783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 694/1404] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:26.290894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d545c97a-9663-42c0-a0df-5c3dec2517da None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:26.291949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d545c97a-9663-42c0-a0df-5c3dec2517da None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:26.292252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d545c97a-9663-42c0-a0df-5c3dec2517da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:26.371163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d545c97a-9663-42c0-a0df-5c3dec2517da None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:26.372467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d545c97a-9663-42c0-a0df-5c3dec2517da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:26.498675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 711/1405] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:26.790077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:26.791650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:26.814594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:26.960137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18f1ea8-b4f3-4f1b-ad49-779f436b370d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:27.066578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:27.100899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:27.101640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:27.101985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:27.102261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:27.103925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:27.207289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:27.209705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c9a2ca-485f-40d4-bb1f-c56a5f45c937 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:27.229879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 695/1406] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:29.453632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:29.454167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:29.454424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:29.455965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 712/1407] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:28:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:29.461491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adff3df-051c-4464-b71f-4a56a94d6f92 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:29.461949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adff3df-051c-4464-b71f-4a56a94d6f92 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:29.462033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adff3df-051c-4464-b71f-4a56a94d6f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:29.539528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9adff3df-051c-4464-b71f-4a56a94d6f92 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:29.540739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9adff3df-051c-4464-b71f-4a56a94d6f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:29.599428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:29.656986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 696/1408] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:28:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:29.954425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:29.956704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:29.971989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:30.147904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9626a9c9-4779-49c2-a9c8-a3192e4ab3bb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:30.258717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:30.303053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:30.303670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:30.303965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:30.304203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:30.305862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:30.425411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:30.427298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f58d8c-700d-493c-b4cd-3d45d5733760 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:30.447537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 713/1409] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:39.655187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=20 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:40.445251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=58 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:44.287723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb71c6-eb70-4ad3-89bc-5b54b4f2992d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:44.288342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb71c6-eb70-4ad3-89bc-5b54b4f2992d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:44.288342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb71c6-eb70-4ad3-89bc-5b54b4f2992d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:44.289513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 697/1410] 10.222.0.16 () {54 vars in 918 bytes} [Tue Oct 4 08:28:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:44.294916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba9e78b-7a58-40cf-963f-0e93dd04a20a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:44.295383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba9e78b-7a58-40cf-963f-0e93dd04a20a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:44.295504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba9e78b-7a58-40cf-963f-0e93dd04a20a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:44.375253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eba9e78b-7a58-40cf-963f-0e93dd04a20a None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:44.376744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eba9e78b-7a58-40cf-963f-0e93dd04a20a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:44.483475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 714/1411] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Oct 4 08:28:44 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:44.498139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70797b52-4234-4b73-826a-fdfd9cb88452 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:44.498416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70797b52-4234-4b73-826a-fdfd9cb88452 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:44.498719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70797b52-4234-4b73-826a-fdfd9cb88452 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:44.500020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 698/1412] 10.222.0.16 () {54 vars in 918 bytes} [Tue Oct 4 08:28:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:44.505071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61236e73-4629-4696-b62f-5368eb96dff2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:44.505353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61236e73-4629-4696-b62f-5368eb96dff2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:44.505625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61236e73-4629-4696-b62f-5368eb96dff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:44.577326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:44.580396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-61236e73-4629-4696-b62f-5368eb96dff2 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:44.582208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-61236e73-4629-4696-b62f-5368eb96dff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:44.698963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 715/1413] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Oct 4 08:28:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:47.388580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:47.389115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:47.389115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:47.390442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 699/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:28:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:47.396432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eaa811-690f-4829-afff-bc290358265d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:47.396873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eaa811-690f-4829-afff-bc290358265d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:47.396999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eaa811-690f-4829-afff-bc290358265d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:47.462803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-27eaa811-690f-4829-afff-bc290358265d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:47.463999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-27eaa811-690f-4829-afff-bc290358265d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:47.518059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:47.562107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 716/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:28:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:28:47.857828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:47.859383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:48.009198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0659336-7e5f-4e94-a01f-6c396893296d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:48.082126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:48.123410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:48.123931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:48.124203 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:48.124465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:48.126400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:48.220111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:48.221713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c74602b-d412-4280-baeb-f7b2c6585be9 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:48.241317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 700/1416] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:49.409694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:49.410225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:49.410225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:49.412233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 717/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:49.418853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7aacc-de54-4aa9-8ea8-1763840dc34c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:49.419315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7aacc-de54-4aa9-8ea8-1763840dc34c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:49.419404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7aacc-de54-4aa9-8ea8-1763840dc34c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:49.496417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-92a7aacc-de54-4aa9-8ea8-1763840dc34c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:49.497626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-92a7aacc-de54-4aa9-8ea8-1763840dc34c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:49.524977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:49.595387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 701/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:28:49.744772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:49.746216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:49.893041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617242b8-1f0e-4402-96f7-4006846ad6e3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:49.937132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:50.009020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:50.009584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:50.009861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:50.010128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:50.011752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:50.101854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:50.103480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeaaddd-05c2-4c3c-9135-49dcddbba2e5 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:50.121238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 718/1419] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:51.277675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:51.278225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:51.278225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:51.279673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 702/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:51.287856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85635f3f-0675-4d0b-bb7a-89d3ec02118a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:51.288340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85635f3f-0675-4d0b-bb7a-89d3ec02118a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:51.288453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85635f3f-0675-4d0b-bb7a-89d3ec02118a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:51.344627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:51.367861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-85635f3f-0675-4d0b-bb7a-89d3ec02118a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:51.369082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-85635f3f-0675-4d0b-bb7a-89d3ec02118a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:51.463930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 719/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:51.628703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:51.632354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:51.783773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f41afd-c400-4275-8f03-56c8959bbc9b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:51.796359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:51.898687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:51.899309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:51.899584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:51.899883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:51.901603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:51.996946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:51.998529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682840df-158a-4408-b10a-e0c0abceaeeb service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:52.016850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 703/1422] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:53.163976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:53.164479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:53.164583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:53.165858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 720/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:53.172299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb79474-9751-4dda-a156-fb69f6ae1a85 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:53.172745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb79474-9751-4dda-a156-fb69f6ae1a85 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:53.172844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb79474-9751-4dda-a156-fb69f6ae1a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:53.193146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:53.252324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6bb79474-9751-4dda-a156-fb69f6ae1a85 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:53.253527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6bb79474-9751-4dda-a156-fb69f6ae1a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:53.360717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 704/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:53.511385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:53.512871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:53.626232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:53.663056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247d3fa-0812-4cb5-b29d-534846595041 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:53.778082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:53.778642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:53.778924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:53.779195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:53.780841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:53.858076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:53.872500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:53.874120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179cd103-7920-4b7a-a0c3-e97b4cc986e8 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:53.892765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 721/1425] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:55.012212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:55.012786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:55.012786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:55.014067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 705/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:55.020994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c530c-f22f-4876-8e70-3b75dd94f190 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:55.021465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c530c-f22f-4876-8e70-3b75dd94f190 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:55.021570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c530c-f22f-4876-8e70-3b75dd94f190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:55.098782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-840c530c-f22f-4876-8e70-3b75dd94f190 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:55.099991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-840c530c-f22f-4876-8e70-3b75dd94f190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:55.197405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 722/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:55.350027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:55.351475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:55.436617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:55.498563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b212f753-b77b-4115-8196-8b0f2b2af3ef None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:55.609679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:55.610199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:55.610478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:55.610778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:55.612389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:55.664111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:55.705631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:55.707824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d362f669-cecf-49f0-a7a0-1b0491a7b49b service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:55.727174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 706/1428] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:56.787313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:56.787893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:56.787893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:56.789214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 723/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:56.795983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f5d65-7400-40bd-aec7-5cc872419f90 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:56.796464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f5d65-7400-40bd-aec7-5cc872419f90 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:56.796557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f5d65-7400-40bd-aec7-5cc872419f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:56.873736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b86f5d65-7400-40bd-aec7-5cc872419f90 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:56.875123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b86f5d65-7400-40bd-aec7-5cc872419f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:56.972242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 707/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:57.127621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:57.129121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:57.178785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:57.274924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988fd77b-26b5-4282-90c1-de6fa1f3d02d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:57.387459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:57.387979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:57.388251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:57.388515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:57.390083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:57.396987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:57.484622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:57.486240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018ad68b-f904-44a2-a070-b5fea73aa559 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:57.505738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 724/1431] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:28:58.586090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:58.586657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:58.586657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:58.587977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 708/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:28:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:28:58.594444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41616bc-302e-4638-bb7f-34e2fa57291c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:58.594923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41616bc-302e-4638-bb7f-34e2fa57291c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:58.595019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41616bc-302e-4638-bb7f-34e2fa57291c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:58.673536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e41616bc-302e-4638-bb7f-34e2fa57291c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:28:58.674811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e41616bc-302e-4638-bb7f-34e2fa57291c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:58.748044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:58.779013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 725/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:28:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:28:58.933591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:28:58.935120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:58.952620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:59.083428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a026c-52c4-439e-a2f0-9f349ef8d112 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:28:59.173115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:28:59.199416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:28:59.199940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:28:59.200210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:28:59.200474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:28:59.202051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:28:59.292511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:28:59.294116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c45a56-444d-4713-812c-82d8b95800ad service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:28:59.313143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 709/1434] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:28:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:00.388732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:00.389316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:00.389316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:00.390637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 726/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:29:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:00.397575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92284ee-9beb-494e-ad22-108307975b43 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:00.398057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92284ee-9beb-494e-ad22-108307975b43 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:00.398164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92284ee-9beb-494e-ad22-108307975b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:00.486046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c92284ee-9beb-494e-ad22-108307975b43 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:00.487298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c92284ee-9beb-494e-ad22-108307975b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:00.528853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:00.585521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 710/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:29:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:00.737119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:00.738862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:00.886967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c025de-f787-4ed3-99f4-cb7253743a40 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:00.941235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:01.001321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:01.001832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:01.002110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:01.002375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:01.004291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:01.094387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:01.096014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bafd53-6ed6-4e53-9685-ade76265f722 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:01.114902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 727/1437] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:02.189505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:02.190049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:02.190049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:02.191990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 711/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:29:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:02.199632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e857d4-11d6-480e-b041-3ef5f43671c5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:02.200184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e857d4-11d6-480e-b041-3ef5f43671c5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:02.200184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e857d4-11d6-480e-b041-3ef5f43671c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:02.280252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-35e857d4-11d6-480e-b041-3ef5f43671c5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:02.281384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-35e857d4-11d6-480e-b041-3ef5f43671c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:02.283525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:02.377984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 728/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:29:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:02.533933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:02.535575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:02.693433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b004043-a8e9-4b83-a003-c942cc4e89be None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:02.720977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:02.818077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:02.818630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:02.818911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:02.819178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:02.820810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:02.916538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:02.918202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4076eab-b30b-4663-b04e-faf6a988e5d4 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:02.937934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 712/1440] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:05.070685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:05.071217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:05.071310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:05.072621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 729/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:05.077907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b4f80-8af7-49b1-97c4-977a45b6badf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:05.078386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b4f80-8af7-49b1-97c4-977a45b6badf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:05.078501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6b4f80-8af7-49b1-97c4-977a45b6badf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:05.119345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:05.150344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fc6b4f80-8af7-49b1-97c4-977a45b6badf None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:05.151603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fc6b4f80-8af7-49b1-97c4-977a45b6badf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:05.368882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 713/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:05.660191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:05.661737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:05.770074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab0b9e-60e9-4a83-9cc0-3b7ce7dcd7f2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:05.806017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:05.848644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:05.849243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:05.849518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:05.849786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:05.851423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:05.962843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:05.964201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30daed6-4e85-41b7-8e37-0ff4b46fbd3f service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:05.982937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 730/1443] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:08.388180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:08.388753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:08.388753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:08.390080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 714/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:08.395584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320aecfb-fea7-4db7-b2d1-ef0ae30562ed None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:08.396002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320aecfb-fea7-4db7-b2d1-ef0ae30562ed None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:08.396155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320aecfb-fea7-4db7-b2d1-ef0ae30562ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:08.464462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-320aecfb-fea7-4db7-b2d1-ef0ae30562ed None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:08.465658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-320aecfb-fea7-4db7-b2d1-ef0ae30562ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:08.478825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:08.567340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 731/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:08.864604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:08.866142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:08.975874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8c83a5-6ae5-48b0-aec0-567cb29ab559 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:08.989393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:09.050523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:09.051160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:09.051433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:09.051698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:09.053321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:09.172028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:09.173485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0c6f64-9212-4080-b1b1-17834aa9a89d service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:09.191933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 715/1446] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:11.697992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09744324-d494-4b5c-b907-260783325bf3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:11.698541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09744324-d494-4b5c-b907-260783325bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:11.698541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09744324-d494-4b5c-b907-260783325bf3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:11.700002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 732/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:11.705905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb2d99-c848-40a7-bbfa-06ed14d825c6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:11.707018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb2d99-c848-40a7-bbfa-06ed14d825c6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:11.707018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb2d99-c848-40a7-bbfa-06ed14d825c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:11.755574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:11.774458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c6eb2d99-c848-40a7-bbfa-06ed14d825c6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:11.775705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c6eb2d99-c848-40a7-bbfa-06ed14d825c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:11.871087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 716/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:12.162403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09744324-d494-4b5c-b907-260783325bf3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:12.163973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-09744324-d494-4b5c-b907-260783325bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:12.267758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09744324-d494-4b5c-b907-260783325bf3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:12.307771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:12.342698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:12.343296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:12.343568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:12.343832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:12.345437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:12.454395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:12.455867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dfa9c3-046a-4e4a-a5db-75e223529119 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:12.475138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 733/1449] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:14.864805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69386606-a679-418f-9b33-d2300e870200 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:14.865439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69386606-a679-418f-9b33-d2300e870200 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:14.865439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69386606-a679-418f-9b33-d2300e870200 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:14.866672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 717/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:14.872080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c23ea-284b-4c49-8f1f-4eabe02b88ad None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:14.872538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c23ea-284b-4c49-8f1f-4eabe02b88ad None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:14.872650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c23ea-284b-4c49-8f1f-4eabe02b88ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:14.948131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-330c23ea-284b-4c49-8f1f-4eabe02b88ad None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:14.949388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-330c23ea-284b-4c49-8f1f-4eabe02b88ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:14.971081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:15.050663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 734/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:15.342818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69386606-a679-418f-9b33-d2300e870200 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:15.344481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69386606-a679-418f-9b33-d2300e870200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:15.450446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69386606-a679-418f-9b33-d2300e870200 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:15.473506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:15.528984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:15.529591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:15.529874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:15.530139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:15.531780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:15.671469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:15.674125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f44c1a3-9325-4958-8883-4114c44308a3 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:15.708193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 718/1452] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:17.988516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:17.989093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:17.989093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:17.990422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 735/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:17.995994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce24aa-4442-453a-bd83-bd746e1d290a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:17.996415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce24aa-4442-453a-bd83-bd746e1d290a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:17.996559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ce24aa-4442-453a-bd83-bd746e1d290a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:18.061839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:18.064603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-12ce24aa-4442-453a-bd83-bd746e1d290a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:18.067295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-12ce24aa-4442-453a-bd83-bd746e1d290a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:18.168634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 719/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:18.464875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:18.466410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:18.572822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21c3ae4-86b3-4727-b1dd-889218d71c5d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:18.628798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:18.657641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:18.660505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:18.660505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:18.660505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:18.661863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:18.787728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:18.789142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef06070-993d-41e8-a608-eb7615c361e8 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:18.808419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 736/1455] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:21.052430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:21.053024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:21.053024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:21.054461 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 720/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:21.060366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4e41-8211-441e-914d-fe233d459024 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:21.061103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4e41-8211-441e-914d-fe233d459024 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:21.061103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a4e41-8211-441e-914d-fe233d459024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:21.130922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-719a4e41-8211-441e-914d-fe233d459024 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:21.132141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-719a4e41-8211-441e-914d-fe233d459024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:21.163914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:21.236754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 737/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:21.535112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:21.536639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:21.652867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b18b6e-2d47-4987-bd73-db75c88481a7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:21.681928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:21.730028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:21.730670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:21.730944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:21.731231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:21.732811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:21.842250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:21.843617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515ee3f3-4f3a-4c7e-99a9-63cf66b4f572 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:21.861112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 721/1458] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:24.394228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:24.394879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:24.394879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:24.396169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 738/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:24.401814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfd7b60-fa8c-44c0-8df2-87452a2847d4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:24.402242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfd7b60-fa8c-44c0-8df2-87452a2847d4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:24.402385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfd7b60-fa8c-44c0-8df2-87452a2847d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:24.473392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dbfd7b60-fa8c-44c0-8df2-87452a2847d4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:24.474631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dbfd7b60-fa8c-44c0-8df2-87452a2847d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:24.483278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:24.582301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 722/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:29:24.866706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:24.868240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:24.976171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade81856-00f4-49ce-a30d-6f110e342dd0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:25.032185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:25.051110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:25.051707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:25.051981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:25.052247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:25.053849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:25.160898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:25.162342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c782a558-1607-4713-954f-a72f8da17b61 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:25.180462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 739/1461] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:27.404837 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:27.405375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:27.405375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:27.406800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 723/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:27.412685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a1dfb-1b55-476d-b914-bcc6bac2aa22 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:27.413189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a1dfb-1b55-476d-b914-bcc6bac2aa22 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:27.413296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3a1dfb-1b55-476d-b914-bcc6bac2aa22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:27.478396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ec3a1dfb-1b55-476d-b914-bcc6bac2aa22 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:27.479622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ec3a1dfb-1b55-476d-b914-bcc6bac2aa22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:27.517560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:27.574542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 740/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:27.858091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:27.859654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:27.961379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd72d88-35a6-45e1-9078-fc0764e0872f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:27.998906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:28.036077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:28.036679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:28.036955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:28.037223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:28.038851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:28.150086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:28.151619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932ac60a-1412-424a-986b-1dc1eb399d46 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:28.172405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 724/1464] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:30.398069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:30.398643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:30.398643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:30.400017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 741/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:30.405482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a368d-2e34-4c29-b696-162ff3bfcfbc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:30.405912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a368d-2e34-4c29-b696-162ff3bfcfbc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:30.406050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a368d-2e34-4c29-b696-162ff3bfcfbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:30.475831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-637a368d-2e34-4c29-b696-162ff3bfcfbc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:30.477079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-637a368d-2e34-4c29-b696-162ff3bfcfbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:30.497136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:30.576697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 725/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:30.866044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:30.867628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:30.969553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fac1256-6311-411e-9e0b-7eaa03ad7c8e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:31.036840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:31.043997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:31.044599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:31.044873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:31.045141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:31.046744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:31.155870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:31.157313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d80cb-14df-4faa-864c-f5257537b6d2 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:31.176162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 742/1467] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:33.399660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:33.400197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:33.400197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:33.401565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 726/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:29:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:33.406982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e885e-8247-4dcd-bd39-e3ed82c0102c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:33.407395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e885e-8247-4dcd-bd39-e3ed82c0102c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:33.407511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e885e-8247-4dcd-bd39-e3ed82c0102c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:33.475736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e98e885e-8247-4dcd-bd39-e3ed82c0102c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:33.476962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e98e885e-8247-4dcd-bd39-e3ed82c0102c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:33.524902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:33.573445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 743/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:29:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:33.860488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:33.862234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:33.965978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d22ceb8-ee9e-402b-bb42-84dfc1118b2d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:34.014426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:34.039879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:34.040485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:34.040773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:34.041025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:34.042638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:34.154136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:34.155624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221a156c-a7a0-4eb5-8445-21e475e429fa service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:34.173260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 727/1470] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:43.572014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=22 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:44.171022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=66 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:54.506266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411dcc8-6a0d-4998-b9f6-f2f58171ad36 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:54.506918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411dcc8-6a0d-4998-b9f6-f2f58171ad36 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:54.507106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411dcc8-6a0d-4998-b9f6-f2f58171ad36 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:54.508464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 744/1471] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:29:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:54.513841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a9b172-9e5f-4f50-8e4f-1ad8ee8bd735 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:54.514301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a9b172-9e5f-4f50-8e4f-1ad8ee8bd735 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:54.514409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a9b172-9e5f-4f50-8e4f-1ad8ee8bd735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:54.589344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-43a9b172-9e5f-4f50-8e4f-1ad8ee8bd735 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:54.590501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43a9b172-9e5f-4f50-8e4f-1ad8ee8bd735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:54.692762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 728/1472] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:29:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:55.013104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:55.013383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:55.013655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:55.015050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 745/1473] 10.222.0.16 () {54 vars in 962 bytes} [Tue Oct 4 08:29:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:55.020996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c086f-a498-4ec7-b4ab-7448c3fbaa30 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:55.021287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c086f-a498-4ec7-b4ab-7448c3fbaa30 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:55.021559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c086f-a498-4ec7-b4ab-7448c3fbaa30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:55.103875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fe9c086f-a498-4ec7-b4ab-7448c3fbaa30 None None] MFA Rules not processed for user `bbc7631562d645dd9b7876a59de53a31`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:55.105081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe9c086f-a498-4ec7-b4ab-7448c3fbaa30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:55.127579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:55.186975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 729/1474] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Oct 4 08:29:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:29:55.195655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:55.197213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:55.307695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f1a2f-bce0-40a5-9b81-a62ea21a69aa None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:55.385804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:55.386414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:55.386730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:55.387001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:55.388619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:55.444827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:55.500034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:55.501553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a692c7-ef10-492b-aac4-9b0461e9c783 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:55.519903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 746/1475] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:29:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:29:57.759856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:57.760449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:57.760550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:57.762118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 730/1476] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:29:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:29:57.768151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1bd9b9-b6b0-405e-b729-cfd0cbd2dd84 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:57.768597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1bd9b9-b6b0-405e-b729-cfd0cbd2dd84 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:57.768776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1bd9b9-b6b0-405e-b729-cfd0cbd2dd84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:57.837003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5e1bd9b9-b6b0-405e-b729-cfd0cbd2dd84 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:29:57.838215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5e1bd9b9-b6b0-405e-b729-cfd0cbd2dd84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:57.934706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 747/1477] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:29:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:29:58.219049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:29:58.221443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:29:58.348090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cea4dcc-58f7-4ef9-b985-ea9c198a618a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:29:58.397572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:29:58.442194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:29:58.442835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:29:58.443109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:29:58.443378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:29:58.444023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:29:58.445753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:29:58.445753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:29:58.445753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:29:58.446934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c897d3c-9961-4943-ad2e-4300447024c2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:29:58.466872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 731/1478] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Oct 4 08:29:58 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:30:00.736454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:00.736997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:00.736997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:00.738352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 748/1479] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:30:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:30:00.743835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9fc163-ad8e-428f-8df9-05a077e173d7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:00.744263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9fc163-ad8e-428f-8df9-05a077e173d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:00.744364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9fc163-ad8e-428f-8df9-05a077e173d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:00.818165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1f9fc163-ad8e-428f-8df9-05a077e173d7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:30:00.819431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1f9fc163-ad8e-428f-8df9-05a077e173d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:00.918962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 732/1480] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:30:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:30:01.218990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:30:01.220950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:01.230586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:01.376845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1485b2-fb5f-42c8-baac-f932ecc683c6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:30:01.463697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:01.499544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:30:01.500437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:01.500712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:01.501035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:01.503547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:01.598081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:30:01.599719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f716d971-f77b-44f8-94cf-499efd3bb341 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:30:01.623714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 749/1481] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:30:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:30:04.037185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:04.037755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:04.037755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:04.039988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 733/1482] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:30:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:30:04.045474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2b463-dd98-467c-a3be-9569ed3bf711 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:04.045961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2b463-dd98-467c-a3be-9569ed3bf711 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:04.046071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f2b463-dd98-467c-a3be-9569ed3bf711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:04.119267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a1f2b463-dd98-467c-a3be-9569ed3bf711 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:30:04.120502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a1f2b463-dd98-467c-a3be-9569ed3bf711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:04.195174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:04.263456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 750/1483] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:30:04.578171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:30:04.579777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:04.608003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:04.698920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2db3577-9663-46bc-a254-798ae6315eec None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:30:04.773665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:30:04.774336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:04.774762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:04.775088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:04.776773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:04.855420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:04.891045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:30:04.893031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0e5d9c-2139-49b9-9aa1-1dc5f3b252c7 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:30:04.911793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 734/1484] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:30:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:30:07.190404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:07.190997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:07.190997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:07.192332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 751/1485] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:30:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:30:07.197891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d5f28-32f7-4cb2-b601-7cd411b479ad None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:07.198353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d5f28-32f7-4cb2-b601-7cd411b479ad None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:07.198462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759d5f28-32f7-4cb2-b601-7cd411b479ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:07.267785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-759d5f28-32f7-4cb2-b601-7cd411b479ad None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:30:07.269001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-759d5f28-32f7-4cb2-b601-7cd411b479ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:07.370758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 735/1486] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:30:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:30:07.667940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:30:07.669485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:07.773868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4b906-2734-47a5-bceb-bc0ed85d1ff6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:30:07.849936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:07.852267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:30:07.853099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:07.853780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:07.854416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:07.856465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:08.116510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:30:08.117971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7722fd88-e3d5-43c4-afb7-299f472dbbf8 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:30:08.139571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 752/1487] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:30:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:30:10.614658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:10.615225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:10.615225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:10.616524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 736/1488] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:30:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:30:10.621912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3c69d-fd95-48b3-aa83-c64efe83bd9e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:10.622334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3c69d-fd95-48b3-aa83-c64efe83bd9e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:10.622436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3c69d-fd95-48b3-aa83-c64efe83bd9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:10.692477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-84a3c69d-fd95-48b3-aa83-c64efe83bd9e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:30:10.693680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84a3c69d-fd95-48b3-aa83-c64efe83bd9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:10.747971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:10.792499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 753/1489] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:30:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:30:11.095692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:30:11.097231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:11.136225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:11.216521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278c72a-2f27-4932-a38e-e539baef61b9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:30:11.302481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:30:11.303115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:11.303384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:11.303650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:11.305241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:11.389259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:11.421428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:30:11.424174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b0f7ee-5581-4926-86db-62a1172aba37 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:30:11.443304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 737/1490] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:30:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:30:13.895333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:13.895892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:13.895892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:13.897223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 754/1491] 10.222.0.16 () {54 vars in 927 bytes} [Tue Oct 4 08:30:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:30:13.903050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef7e37-182a-41df-a2a2-d35518302008 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:13.903602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef7e37-182a-41df-a2a2-d35518302008 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:13.903602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef7e37-182a-41df-a2a2-d35518302008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:13.972734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7eef7e37-182a-41df-a2a2-d35518302008 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:30:13.973956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7eef7e37-182a-41df-a2a2-d35518302008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:14.078239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 738/1492] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Oct 4 08:30:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:30:14.370279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:30:14.372923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:14.482999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b67d7e-cc54-4472-b646-98f9592029b4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:30:14.560531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:30:14.561132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:30:14.561408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:30:14.561683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:30:14.564180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:30:14.573508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:14.686431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:30:14.689077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bfeefb-8d7a-4119-93fd-5fed91cc8f9e service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:30:14.718774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 755/1493] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:30:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:30:24.076242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=82 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:30:24.713922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=50 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:29.987600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:29.988202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:29.988202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:30.070457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:30.071699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:30.169138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 739/1494] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:31:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:31:30.177582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:30.178081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:30.178188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:30.254967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:30.256102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:30.352119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 756/1495] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:31:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:31:30.356759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:30.358668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:30.370027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add04b8-7670-4ea2-bd7a-1dcfaf1d6810 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:30.430517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:30.464860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:30.465458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:30.465733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:30.466000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:31:30.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:30.480845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 740/1496] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:31:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:31:30.489732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:30.499810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7dcc7-0457-4057-ae59-51d135fe0188 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:30.566169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:30.599519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:30.600119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:30.600395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:30.600665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:30.601290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:30.603144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:30.603144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:31:30.603144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:31:30.604336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:30.659290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 757/1497] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:31:30 2022] POST /identity/v3/projects => generated 379 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:31:30.663486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:30.664998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:30.699080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698791c6-defc-448c-8ec4-b25d7342c82b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:30.793539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:30.794144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:30.795510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:30.795510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:30.796205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:30.797353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:30.797353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:31:30.797353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:31:30.798884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:30.846478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 741/1498] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:31:30 2022] POST /identity/v3/users => generated 290 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:31:30.851015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:30.852848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:30.919177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a70f66-ec3a-48d6-b6ef-f381e841d47d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:31.006680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:31.019370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:31.019976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.020248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.020514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.021155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:31.022171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:31.022171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:31:31.022171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:31:31.023083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:31.036414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 758/1499] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:31:30 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:31:31.040492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:31.042007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:31.053787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b66de2-1b65-47b3-a9dc-9f3eb103defd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:31.083992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:31.165047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:31.165653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.165923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.166194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.166871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:31.167875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:31.167875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:31:31.167875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:31:31.168752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804fbd1-a2b1-44f0-a66c-8d83b2b0bb8a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:31.181661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 742/1500] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:31:31 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:31:31.186188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:31.187748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:31.221859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7116f0f-589e-4ede-be66-bde4eac5c68e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:31.317596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:31.318261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.318534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.318830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] PATH_INFO: `/v3/projects/d6507cb141f44208b252ea0d6aba19ae/users/bb0f820988814286a28301592a17c327/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.344340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6507cb141f44208b252ea0d6aba19ae, user_id=bb0f820988814286a28301592a17c327, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:31.345392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:31.345392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:31:31.345392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:31:31.347367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaade071-7dd0-466f-93b8-e23f87d6340a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:31.375766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 759/1501] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:31:31 2022] PUT /identity/v3/projects/d6507cb141f44208b252ea0d6aba19ae/users/bb0f820988814286a28301592a17c327/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:31:31.388895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c44cfb-f6a2-48c9-8736-b61a0451aa70 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.389411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c44cfb-f6a2-48c9-8736-b61a0451aa70 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.389805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c44cfb-f6a2-48c9-8736-b61a0451aa70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.453316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e1c44cfb-f6a2-48c9-8736-b61a0451aa70 None None] MFA Rules not processed for user `bb0f820988814286a28301592a17c327`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:31.454481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e1c44cfb-f6a2-48c9-8736-b61a0451aa70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:31.540012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 743/1502] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:31:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:31:31.558648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.559105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.559553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.562452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 760/1503] 10.222.0.16 () {54 vars in 963 bytes} [Tue Oct 4 08:31:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:31:31.567859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d90b31-fff9-428b-ac47-b9c58638ec53 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.568139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d90b31-fff9-428b-ac47-b9c58638ec53 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.568409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d90b31-fff9-428b-ac47-b9c58638ec53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.605091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:31.655654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-69d90b31-fff9-428b-ac47-b9c58638ec53 None None] MFA Rules not processed for user `546c67695c1e447da68b91637aefe135`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:31.656888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69d90b31-fff9-428b-ac47-b9c58638ec53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:31.737525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 744/1504] 10.222.0.16 () {58 vars in 1060 bytes} [Tue Oct 4 08:31:31 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:31:31.745564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:31.747165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:31.807337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:31.866053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53afade6-0ccf-4973-8701-d45a31039991 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:31.955331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:31.956669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:31.957203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:31.957591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:31.960023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:32.087275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:32.088650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:32.092554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:32.108157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 761/1505] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:31:31 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:31:32.160300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701043bb-1277-4e42-a636-d55b6490ec53 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:32.160757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701043bb-1277-4e42-a636-d55b6490ec53 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:32.160867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701043bb-1277-4e42-a636-d55b6490ec53 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:32.162100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 745/1506] 10.222.0.16 () {52 vars in 830 bytes} [Tue Oct 4 08:31:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:31:32.166120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:32.167642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:32.269092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1054918e-6651-4f67-8fde-30d999be0397 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:32.374704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:32.375265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:32.375541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:32.375809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] PATH_INFO: `/v3/users/bb0f820988814286a28301592a17c327` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:32.391902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb0f820988814286a28301592a17c327)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:32.392952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:32.392952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:31:32.392952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:31:32.394187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:32.406358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:32.471516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb0f820988814286a28301592a17c327'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:32.488373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb0f820988814286a28301592a17c327'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:32.513760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb0f820988814286a28301592a17c327'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:32.514239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb0f820988814286a28301592a17c327'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:32.524424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb0f820988814286a28301592a17c327'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:32.532789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 762/1507] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:31:32 2022] DELETE /identity/v3/users/bb0f820988814286a28301592a17c327 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:31:32.537297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:32.537578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:32.537879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:32.611714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:31:32.612950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:32.695890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:32.708684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 746/1508] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:31:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:31:32.718795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:32.720039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:32.833771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf17269-d570-44ed-ba48-f00a73fe7ec4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:32.892343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:32.914324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:32.914967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:32.915245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:32.915516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:32.917120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:33.036473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:33.039146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35f949e-9429-422a-bde7-e3d7d73c559a service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:33.062239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 763/1509] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:31:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:31:33.458735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:31:33.459980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:31:33.471933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78e9bd-63f4-4dca-a237-0e91f888cce7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:31:33.573629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:31:33.574264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:31:33.574537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:31:33.574836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] PATH_INFO: `/v3/projects/d6507cb141f44208b252ea0d6aba19ae` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:31:33.585240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6507cb141f44208b252ea0d6aba19ae)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:31:33.586266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:31:33.586266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:31:33.586266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:31:33.587514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:31:33.648863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6507cb141f44208b252ea0d6aba19ae', 'request_id': 'req-5b80e9b4-4a90-4212-8946-4e1887212f2c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:33.663793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6507cb141f44208b252ea0d6aba19ae', 'request_id': 'req-5b80e9b4-4a90-4212-8946-4e1887212f2c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:33.678012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6507cb141f44208b252ea0d6aba19ae', 'request_id': 'req-5b80e9b4-4a90-4212-8946-4e1887212f2c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:33.704524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] The token cache is being invalidate because project d6507cb141f44208b252ea0d6aba19ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:31:33.704815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:33.705357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5b80e9b4-4a90-4212-8946-4e1887212f2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:31:33.706774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 747/1510] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:31:33 2022] DELETE /identity/v3/projects/d6507cb141f44208b252ea0d6aba19ae => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:31:43.059117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=62 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:31:43.701817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:06.266118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.267084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.267084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.268194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 764/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:33:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:06.274688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c87bbe-dbb9-4de0-82ce-afc364980cde None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.275478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c87bbe-dbb9-4de0-82ce-afc364980cde None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.275623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c87bbe-dbb9-4de0-82ce-afc364980cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.372453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-33c87bbe-dbb9-4de0-82ce-afc364980cde None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:06.374086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-33c87bbe-dbb9-4de0-82ce-afc364980cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:06.522937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 748/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:33:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:06.700532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:06.703882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:06.861803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15521c-d5c6-4c8a-8797-95ae2fd98555 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:06.868580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:06.869697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:06.870067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:06.870454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:06.880537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:06.882213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1048d095-b938-4f9e-bb3c-3b371e817a2d service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:06.900860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 765/1513] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:33:07.836115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24608a5a-3e43-49fd-94a7-24a10f183a00 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.836775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24608a5a-3e43-49fd-94a7-24a10f183a00 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.837117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24608a5a-3e43-49fd-94a7-24a10f183a00 None None] PATH_INFO: `/v3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.839654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 749/1514] 10.222.0.17 () {54 vars in 925 bytes} [Tue Oct 4 08:33:07 2022] GET /identity/v3 => generated 255 bytes in 6 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:33:07.847225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5faa4c-eea1-431c-8fca-d3c418cd9b8b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:07.848040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5faa4c-eea1-431c-8fca-d3c418cd9b8b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:07.848040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5faa4c-eea1-431c-8fca-d3c418cd9b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:07.930799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ef5faa4c-eea1-431c-8fca-d3c418cd9b8b None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:07.932484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ef5faa4c-eea1-431c-8fca-d3c418cd9b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:08.025615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:08.033241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 766/1515] 10.222.0.17 () {58 vars in 1015 bytes} [Tue Oct 4 08:33:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:33:08.148831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c84d9b-24c9-4d71-9d2e-a026cbcaa7f4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.149320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c84d9b-24c9-4d71-9d2e-a026cbcaa7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.149807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c84d9b-24c9-4d71-9d2e-a026cbcaa7f4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.152956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 750/1516] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:33:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:08.160285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e8d49-ec83-4316-a195-edecf4a2b7e9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.160565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e8d49-ec83-4316-a195-edecf4a2b7e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.160842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e8d49-ec83-4316-a195-edecf4a2b7e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.191363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-128e8d49-ec83-4316-a195-edecf4a2b7e9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:08.192691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-128e8d49-ec83-4316-a195-edecf4a2b7e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:08.202533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 767/1517] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:33:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:08.358476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c84d9b-24c9-4d71-9d2e-a026cbcaa7f4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:08.374675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c84d9b-24c9-4d71-9d2e-a026cbcaa7f4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:08.380412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:08.381015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:08.381290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:08.381558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:08.390407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:08.392032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac852a5-936a-4b1d-995d-c868a4bcbb45 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:08.402995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 751/1518] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:33:11.663685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f20f7-1b91-4d72-b2b6-2b5a9805595f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.664574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f20f7-1b91-4d72-b2b6-2b5a9805595f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.664574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f20f7-1b91-4d72-b2b6-2b5a9805595f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.666926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 768/1519] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:33:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:11.672917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90b1fb-e0bc-4a83-b796-d9cf9c8b6cde None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.673536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90b1fb-e0bc-4a83-b796-d9cf9c8b6cde None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.673536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd90b1fb-e0bc-4a83-b796-d9cf9c8b6cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.695491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd90b1fb-e0bc-4a83-b796-d9cf9c8b6cde None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:11.696739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd90b1fb-e0bc-4a83-b796-d9cf9c8b6cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:11.707181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 752/1520] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:33:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:33:11.716419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43f20f7-1b91-4d72-b2b6-2b5a9805595f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:11.738515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43f20f7-1b91-4d72-b2b6-2b5a9805595f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:11.746960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:11.756813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:11.757121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:11.757397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:11.767551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:11.769479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b3e0c4-cf1b-489e-a7a1-266b69b45573 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:11.781439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 769/1521] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:33:19.977720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ed55f-d339-47d9-942b-c552fd02bc51 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.979116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ed55f-d339-47d9-942b-c552fd02bc51 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.979694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97ed55f-d339-47d9-942b-c552fd02bc51 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:19.981222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 753/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:33:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:33:19.988550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975ed6f-3faa-4f55-8fa3-1e3709b2098d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:19.989142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975ed6f-3faa-4f55-8fa3-1e3709b2098d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:19.989142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6975ed6f-3faa-4f55-8fa3-1e3709b2098d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.011109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6975ed6f-3faa-4f55-8fa3-1e3709b2098d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:20.012471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6975ed6f-3faa-4f55-8fa3-1e3709b2098d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:20.020586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 770/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:33:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:33:20.184616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97ed55f-d339-47d9-942b-c552fd02bc51 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:20.206674 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97ed55f-d339-47d9-942b-c552fd02bc51 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:20.212928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:20.213770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:20.214044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:20.214308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:20.224269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:20.226117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d355cf77-8186-4d61-acf0-56bd000b5f15 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:20.237950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 754/1524] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:33:21.400675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd775e-1d45-47f6-949a-c8fabe01e061 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.401392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd775e-1d45-47f6-949a-c8fabe01e061 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.401530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd775e-1d45-47f6-949a-c8fabe01e061 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.402891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 771/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Oct 4 08:33:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:21.413817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75424a9f-df04-4796-a0e4-5ce9ce4bc263 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.414358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75424a9f-df04-4796-a0e4-5ce9ce4bc263 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.414694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75424a9f-df04-4796-a0e4-5ce9ce4bc263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.447667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-75424a9f-df04-4796-a0e4-5ce9ce4bc263 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:21.449734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-75424a9f-df04-4796-a0e4-5ce9ce4bc263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:21.463670 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 755/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Oct 4 08:33:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:21.630342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd775e-1d45-47f6-949a-c8fabe01e061 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:21.645631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd775e-1d45-47f6-949a-c8fabe01e061 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:21.652598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:21.653224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:21.653494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:21.653761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:21.662987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:21.664650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2241c86-b554-41a8-b55f-580dd50b99db service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:21.675790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 772/1527] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:33:25.789608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:25.790227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:25.791046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:25.793546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 756/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:33:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:25.799348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a4df1-7de5-471c-a47c-b084d53fb13e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:25.799871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a4df1-7de5-471c-a47c-b084d53fb13e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:25.799984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a4df1-7de5-471c-a47c-b084d53fb13e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:25.820550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ff5a4df1-7de5-471c-a47c-b084d53fb13e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:25.822463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ff5a4df1-7de5-471c-a47c-b084d53fb13e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:25.830702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 773/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:33:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:33:26.126442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:26.128264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:26.168009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:26.218412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb92c17a-bd26-48dd-ad42-1cf6371861be None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:26.222511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:26.223175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:26.223449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:26.223711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:26.234218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:26.235736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f871e-1d2a-4050-a50e-962455c8f6eb service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:26.247246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 757/1530] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:33:28.886159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fe7e34-0f58-4393-b745-f832a7c7cddd None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.887133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fe7e34-0f58-4393-b745-f832a7c7cddd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.887133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fe7e34-0f58-4393-b745-f832a7c7cddd None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.888906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 774/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:33:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:28.895655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6f347-5ebe-4d25-92a8-8ec83683e921 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:28.896314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6f347-5ebe-4d25-92a8-8ec83683e921 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:28.896314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6f347-5ebe-4d25-92a8-8ec83683e921 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:28.930993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-99b6f347-5ebe-4d25-92a8-8ec83683e921 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:28.932243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-99b6f347-5ebe-4d25-92a8-8ec83683e921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:28.944786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 758/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:29.353545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fe7e34-0f58-4393-b745-f832a7c7cddd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:29.367328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fe7e34-0f58-4393-b745-f832a7c7cddd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:29.371776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:29.372493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:29.372772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:29.373041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:29.387158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:29.391335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fee4b4b-f047-4cea-b0dd-3c80a4ec0624 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:29.403732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 775/1533] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:33:33.517804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.518380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.518380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.519714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 759/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:33:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:33.526206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41705b7-2b05-432b-aed2-d3f53645bb8c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.526769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41705b7-2b05-432b-aed2-d3f53645bb8c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.526769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41705b7-2b05-432b-aed2-d3f53645bb8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.546493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d41705b7-2b05-432b-aed2-d3f53645bb8c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:33.547760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d41705b7-2b05-432b-aed2-d3f53645bb8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:33.557540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 776/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:33:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:33.872102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:33.874292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:33.957646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c07d09-4de4-4b26-9d47-c2cd9cafff8a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:33.961340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:33.961975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:33.962249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:33.962514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:33.971910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:33.973326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98bc77-7289-44ba-99b2-f52da6115ef1 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:33.983945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 760/1536] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:33:36.289632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.290886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.290886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.292825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 777/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Oct 4 08:33:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:33:36.299072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe671a8a-16f4-41ef-9ce4-d28a20143a17 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.299975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe671a8a-16f4-41ef-9ce4-d28a20143a17 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.299975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe671a8a-16f4-41ef-9ce4-d28a20143a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.324789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fe671a8a-16f4-41ef-9ce4-d28a20143a17 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:33:36.326057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe671a8a-16f4-41ef-9ce4-d28a20143a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:36.334307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 761/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Oct 4 08:33:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:33:36.659345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:33:36.661189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:33:36.675026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c498c548-6e3c-4067-b671-686ef37e3ba7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:33:36.679061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:33:36.679734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:33:36.680046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:33:36.680349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:33:36.691461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:33:36.692899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0738bd2-15d2-4020-8a5d-f8c8183c9a65 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:33:36.704493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 778/1539] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:33:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:33:46.317776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=63 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:33:46.699654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=37 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:04.361088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.361849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.361849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.388701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.389906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.405856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 762/1540] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.416451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.417475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.417973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.447163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.448925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.462213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 779/1541] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.467813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.486601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b67ab3-7332-4d42-a510-1304622e6f9b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.496684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.497312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.497583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.497844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.498781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:04.499080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.500692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc136c9-abd7-4398-acc9-032ecb9cb06e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.517934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 763/1542] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:04.528410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.544567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a895da-d7d9-460f-8d35-367077e89892 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.552118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.554055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.554726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.555221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.556085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.558661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.560761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d866f0-22eb-4788-8307-83e2cf624a6b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.566639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d866f0-22eb-4788-8307-83e2cf624a6b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.566639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d866f0-22eb-4788-8307-83e2cf624a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.598646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-70d866f0-22eb-4788-8307-83e2cf624a6b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.599941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-70d866f0-22eb-4788-8307-83e2cf624a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.620849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 764/1543] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:04.632598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b633-650b-4e2e-bccd-ce7f724f353f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.632870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b633-650b-4e2e-bccd-ce7f724f353f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.633131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b633-650b-4e2e-bccd-ce7f724f353f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.651063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 780/1544] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/projects => generated 379 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:04.656074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.665243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b047b633-650b-4e2e-bccd-ce7f724f353f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.666465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b047b633-650b-4e2e-bccd-ce7f724f353f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.770439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 765/1545] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.773091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.773388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.773639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.805953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eef013e-63ca-4788-8c83-bdc30231044e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.806460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.810577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.822281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 766/1546] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.822942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.823375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.824393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.824917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.825394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.826259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.827570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:04.827570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:35:04.827570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:35:04.829220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ee322e-f416-4dd6-a0ac-4b87f3e2a4e9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.840943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1c817-ba31-4616-870e-b4b8cf59e1a9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.853951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.854755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.855030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.855306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.856195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:04.856492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.858310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac0917e-4790-4d60-a88a-c117690d846a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.873214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 767/1547] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:04.875165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.875453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.875724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.876812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 781/1548] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/users => generated 290 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:04.878383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.878932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.879705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.920082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.921216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.928679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:04.936141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:04.943710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 768/1549] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.948980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.958320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 782/1550] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:04.960139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:04.976195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818bc026-085c-43fb-80d2-d086f53f98bd None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.984514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16bb92-c865-4f91-83cc-65db409253eb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:04.986579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.987267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.987548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.987823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.988490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.989563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:04.989563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:35:04.989563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:35:04.995072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:04.995869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:04.996002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:04.996188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:04.996821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:04.997180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0881ae10-26d0-4a68-9938-056f4093eab0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:04.998724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.027577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 783/1551] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:04 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:05.028758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.050128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f0470-4f3e-43f2-ad87-443ee86970d8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.063935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.065047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.066312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.066821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.068709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.071388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.072501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:05.072501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:35:05.072501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:35:05.073234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 769/1552] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:04 2022] POST /identity/v3/projects => generated 389 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.078509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3e2703-1a80-48f5-85ad-fa504361dac8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.079874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.080215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.080486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.096124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 784/1553] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.097467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.097755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.098208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.106468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.107717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.129547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.130786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.196079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 770/1554] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.197294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.209677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f30aea9-958b-432d-adca-adafa3b5231c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.212606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 785/1555] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.213582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.215102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.215691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.216011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.216240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.216892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.218495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb67079-83e1-421d-9f5e-0a53cc78e5d4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.226512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37d00e1-4ba4-466b-be08-13e2a558ca57 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.230414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 771/1556] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:05.231848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99b4eb-72a0-47fc-bfec-88ba36670d70 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.232135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99b4eb-72a0-47fc-bfec-88ba36670d70 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.232426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99b4eb-72a0-47fc-bfec-88ba36670d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.235097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.235719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.235997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.236280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.236907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.238855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.253560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6e99b4eb-72a0-47fc-bfec-88ba36670d70 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.254896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6e99b4eb-72a0-47fc-bfec-88ba36670d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.262670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 786/1557] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/users => generated 295 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.263863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.266633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 772/1558] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.266633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a9a42d-6be1-4664-b12a-5c8d55879706 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.266633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a9a42d-6be1-4664-b12a-5c8d55879706 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.266633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a9a42d-6be1-4664-b12a-5c8d55879706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.269505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:05.274699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f425063-280f-4846-9c4d-994af9b8ac37 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.281046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.281672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.281954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.282259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.283148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.285142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.290230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-33a9a42d-6be1-4664-b12a-5c8d55879706 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.291467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-33a9a42d-6be1-4664-b12a-5c8d55879706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.299666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 773/1559] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.301172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5b4ed5-1636-4d7f-af4d-75349b145c03 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.301425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5b4ed5-1636-4d7f-af4d-75349b145c03 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.301770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5b4ed5-1636-4d7f-af4d-75349b145c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.310276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:05.318787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 787/1560] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.323915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.330750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dc5b4ed5-1636-4d7f-af4d-75349b145c03 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.332086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dc5b4ed5-1636-4d7f-af4d-75349b145c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.402362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 774/1561] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.403823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.404105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.404378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.418306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581e53be-ebbe-4bb9-987d-a263da05837e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.424174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.424813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.425123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.425399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.425525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.426070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.426373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.426731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.428194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.435127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 775/1562] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.435750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.442589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 788/1563] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.443132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.450955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b4b902-91fe-4841-b673-a4349bfd6445 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.454336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254ba754-e363-48ce-b773-63f3881f88ea admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.455718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.456453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.456887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.457161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] PATH_INFO: `/v3/projects/3f34eba992024f6881d1292c9f7cfd48/users/d002a844a71649d884b4ca870eb9d387/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.460054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.460653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.460923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.461195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.461889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.463588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996892b2-d866-4c5a-9c77-e120e1652917 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.474542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f34eba992024f6881d1292c9f7cfd48, user_id=d002a844a71649d884b4ca870eb9d387, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.478003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 789/1564] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:05.478194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:05.478194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:35:05.478194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:35:05.478486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.478486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.478486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.478808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e0343d-a194-49ff-8c1e-b6fe4be5e04b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.492704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 776/1565] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:05 2022] PUT /identity/v3/projects/3f34eba992024f6881d1292c9f7cfd48/users/d002a844a71649d884b4ca870eb9d387/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:05.494152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0451402-bf31-4bea-8b18-e743afdc47f5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.494437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0451402-bf31-4bea-8b18-e743afdc47f5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.494809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0451402-bf31-4bea-8b18-e743afdc47f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.503852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.505167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.517293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c0451402-bf31-4bea-8b18-e743afdc47f5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.518536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c0451402-bf31-4bea-8b18-e743afdc47f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.574732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 790/1566] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.575672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.584905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 777/1567] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.588695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf609786-6d70-420d-a2e3-929075f2553d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.589911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.590201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.590479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.593941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.594568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.594909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.595121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.595752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.597698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.614723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.616501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.625668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 791/1568] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/users => generated 292 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.625898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 778/1569] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.626352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.629741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.636921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b71201-04db-41e2-9784-d8bd59a58717 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.640094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976a1184-fa35-4f54-8af3-d7d2cac5bbf7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.644870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.645159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.645643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.645855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.646015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.646187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.646357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.646610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.647082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.647584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.648019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.649006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0890a8-cae6-4b4a-a23a-6e32cecb2c38 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.649851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.665705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 792/1570] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.667194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.683544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da71dde-aec8-494d-a695-85c9d1b48e0c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.690299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 779/1571] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/projects => generated 371 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.690299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.690299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.692580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.721903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:05.723083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:05.731512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:05.762026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.762762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.763113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.763388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.764055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.765701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.768572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:05.776945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 793/1572] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:05.777647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.787701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2501e2f-c2ba-4b16-8e91-a9d7781ac9df admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.789053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 780/1573] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:05.791466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.799062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.799589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.799865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.800134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.800787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.801090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.801428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3483adf-6e5e-4d17-a654-7273dd014a0b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.802722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.806733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.807338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.807616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.807889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.808536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.808839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.810424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.816967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 794/1574] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.817816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.823537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 781/1575] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.824736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.835434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578a3272-5122-4962-a7ad-0a26a3544d0a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.835758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5e3194-50cc-4806-81fd-85e507f565c4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.840469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.840618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.841089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.841196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.841378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.841493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.841660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.841792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.842302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.842418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:05.842633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.843946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.844234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.854976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 795/1576] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:05.855698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.860384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 782/1577] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:05.860384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.866136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc52276-8035-421c-b12b-7e9044a9612a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.870832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.871078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caad6e58-21e5-4f16-9155-45728442151b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.871439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.871745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.871988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.872634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.874858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.876238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.876880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.877230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.877453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.878084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.880023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:05.904127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 783/1578] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/users => generated 286 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.904459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 796/1579] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:05 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:05.905072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.905452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:05.919359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac36393-f2ab-4eae-a818-6ccf5248122b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.967546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d38fb77-ddcb-4d10-b321-7973a8a53fc4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:05.971201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.971813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.972144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.972365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.977193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:05.978342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:05.978652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:05.978926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] PATH_INFO: `/v3/projects/3e938efbd6804a86946d80ae92e88b16/users/87ca9fc2e013477cad3a3de8e8811274/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:05.994891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e938efbd6804a86946d80ae92e88b16, user_id=87ca9fc2e013477cad3a3de8e8811274, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:05.997556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.013022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 784/1580] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:05 2022] PUT /identity/v3/projects/3e938efbd6804a86946d80ae92e88b16/users/87ca9fc2e013477cad3a3de8e8811274/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:06.013815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.042193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.044225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.058749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 797/1581] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:06.060446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.072290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafd435-6954-41ee-ad41-caceb9146c7d service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.088732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7791-b389-4d57-8e5c-610d05679454 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.094430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.095088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.095366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.095637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.096299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.098235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.111478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:06.131602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 785/1582] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/projects => generated 391 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.135779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.148171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:06.152443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.152443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.152443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.152443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.152878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.155513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.191026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 798/1583] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/projects => generated 373 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.191720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.220863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef590352-8478-40ed-88d4-a7d8fa45b770 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.271201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63578ae-9e3d-4227-8f77-afaadf8dbf64 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.277941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.278900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.279532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.279963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.280624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.283361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.299656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.300673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.301372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.301821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.304034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.306110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.319544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 799/1584] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/projects => generated 393 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.319544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.325407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 786/1585] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.326990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.341039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b149af-a2d6-4198-b8c5-497c8c2e46cf admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.396472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce602000-ea26-47c2-b894-429e6387d888 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.403648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.404716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.405078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.405362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.406003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.408648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158141f2-2eeb-41e2-aa21-169b503e3eb3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.413803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.414429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.415422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.415796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.416544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.418268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.431593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 800/1586] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.433223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.433342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 787/1587] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.433437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.433601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.433828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.445711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba4b295-31ba-407c-8f1e-9ef1381f3031 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.450601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.451212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.451502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.451761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.452452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.454391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.455408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:06.457107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:06.465716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 801/1588] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:06.466689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.477424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cd04ca-2a1b-4584-9703-907158e922fe None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.482750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.483356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.483633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.483899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.484624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 788/1589] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/users => generated 296 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.484756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.486612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.488241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.498094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382bafd5-273c-4099-9c7c-3863d18d78e6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.504655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.505220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.505460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.505732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.506962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.508886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.514022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 802/1590] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/users => generated 287 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.514022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.525282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354e1f3-e0ca-4eee-8aa9-4f87179dbbc0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.531526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.532279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.532565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.532841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] PATH_INFO: `/v3/projects/d61d65375ccf4967b7545338ce72bf68/users/baf5ab01e28944258aa2fdfe7951b504/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.543062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:06.554336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d61d65375ccf4967b7545338ce72bf68, user_id=baf5ab01e28944258aa2fdfe7951b504, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.555950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 789/1591] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/users => generated 297 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:06.556578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.557416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.569885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371a6e0-41a8-4e43-981a-f881e2654605 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.574285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 803/1592] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:06 2022] PUT /identity/v3/projects/d61d65375ccf4967b7545338ce72bf68/users/baf5ab01e28944258aa2fdfe7951b504/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:06.584210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.594345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:06.655354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.655968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.656244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.656512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.657159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.659466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.670422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f760742-70b8-46b7-aa92-7c8962e7d46a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.671571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 790/1593] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.672506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.675201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:06.679960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.680642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.680927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.681308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.682210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.684950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.696346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 804/1594] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.700169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.710780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89affe1-3ee2-49bd-acc3-406af00aab03 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.718202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.718939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.719218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.719767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.721021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.723471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.739680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 805/1595] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.739680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.746070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26e6764-6704-44a9-ad00-63d9c6f72e37 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.750302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe82b02-0913-4225-b38f-a7d71c7f4fe2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.750684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.751810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.752160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.752465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.755648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.756350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.757018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.757312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.757953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.759611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.764406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.766651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1aae1b-974b-456a-8f6f-ad9cff411202 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.774732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 806/1596] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.775849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.779780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 791/1597] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:06.780933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.781215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.781486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.788557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8573d539-3b68-4008-831f-5998447a6fae admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.793797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.794667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.795279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.795279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.795904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.797514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.812774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 807/1598] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.812774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.819029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] MFA Rules not processed for user `baf5ab01e28944258aa2fdfe7951b504`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:06.820477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:06.821775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497d377-dff3-4a01-9fac-f998869a4e85 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.827024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.827639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.827936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.828190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.828839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.830456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.841694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 808/1599] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:06.842662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.855036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba3f643-bd80-494b-b521-18c11834bead admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.863955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:06.864870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:06.865408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:06.865590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] PATH_INFO: `/v3/projects/97144851313240fd838b173918b5a299/users/d9e8091bc8824c29be0a285767a3d32d/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:06.883553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=97144851313240fd838b173918b5a299, user_id=d9e8091bc8824c29be0a285767a3d32d, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:06.886247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:06.900997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 809/1600] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:06 2022] PUT /identity/v3/projects/97144851313240fd838b173918b5a299/users/d9e8091bc8824c29be0a285767a3d32d/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:06.902057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.913908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 792/1601] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:06.915034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:06.926463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d02fb3-723f-46d9-8b50-48f35ef2f553 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:06.983840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.005548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.006430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.006592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071de39e-683f-49a2-9fee-150c3e4eeb8b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.006850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.007176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.007906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.009687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.012565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.013330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.013513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.013797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.014424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.017199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.024598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 793/1602] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:07.024598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.030540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 810/1603] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:06 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:07.031609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.035506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3446001-cc90-4469-b3fe-c0134a83c449 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.041662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.042395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.042774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.043172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.043855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.045708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a5640b-0042-4be2-aabf-ca507ff0be45 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.046839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e1c578-5dab-451b-b3b7-644837a816be admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.060355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 794/1604] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:07 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:07.062051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.063058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.063058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.067078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.068340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.069681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.070843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] PATH_INFO: `/v3/projects/2b940c43c46c402bbc4c3c0461b51061/users/0c1de2efe2ed42508d7ab4a2e9d91e92/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.085496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.094310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b940c43c46c402bbc4c3c0461b51061, user_id=0c1de2efe2ed42508d7ab4a2e9d91e92, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.097645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490ae5ac-1e81-43f7-9429-f944da6d5642 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.111913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] MFA Rules not processed for user `d9e8091bc8824c29be0a285767a3d32d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.113750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:07.136216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 811/1605] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:07 2022] PUT /identity/v3/projects/2b940c43c46c402bbc4c3c0461b51061/users/0c1de2efe2ed42508d7ab4a2e9d91e92/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:07.137958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac21cb13-9add-40d6-b83b-5f2b2fccac87 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.138955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac21cb13-9add-40d6-b83b-5f2b2fccac87 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.142513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac21cb13-9add-40d6-b83b-5f2b2fccac87 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.144186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 812/1606] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:35:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:07.153118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac21cb13-9add-40d6-b83b-5f2b2fccac87 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.269947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 795/1607] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:07.269947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.286436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac21cb13-9add-40d6-b83b-5f2b2fccac87 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.298264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c60c10b-fb0f-44d1-b7e5-91a8eedbb364 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.299778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.302128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.302634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.303641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] PATH_INFO: `/v3/projects/7f158aa9def14740b321dacf2fbacab1/users/2cf05c1f95264293ae851bb86ed6aeea/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.304520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.305200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.305476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.305754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] PATH_INFO: `/v3/projects/33adc74b068441a1b62341362286fe83/users/e7bca9820438452685bbc9db2ee90e02/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.324297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f158aa9def14740b321dacf2fbacab1, user_id=2cf05c1f95264293ae851bb86ed6aeea, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.327343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.335871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] RBAC: Authorizing `identity:create_grant(project_id=33adc74b068441a1b62341362286fe83, user_id=e7bca9820438452685bbc9db2ee90e02, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.346500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be653035-6d36-47e8-b1ca-9139db42f13f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.361291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 813/1608] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:07 2022] PUT /identity/v3/projects/7f158aa9def14740b321dacf2fbacab1/users/2cf05c1f95264293ae851bb86ed6aeea/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:07.362600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.390691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 796/1609] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:07 2022] PUT /identity/v3/projects/33adc74b068441a1b62341362286fe83/users/e7bca9820438452685bbc9db2ee90e02/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:07.393142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1691b7d6-96dc-4927-b7d5-6696dc3c350c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.393428 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1691b7d6-96dc-4927-b7d5-6696dc3c350c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.393702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1691b7d6-96dc-4927-b7d5-6696dc3c350c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.447237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1691b7d6-96dc-4927-b7d5-6696dc3c350c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.449499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1691b7d6-96dc-4927-b7d5-6696dc3c350c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:07.537466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb58ff6-1df9-4695-8b0f-600286d48733 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.608304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 797/1610] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:07.609501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212fd36-0ee7-4708-8ce2-258bccd03ad5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.609792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212fd36-0ee7-4708-8ce2-258bccd03ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.610067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212fd36-0ee7-4708-8ce2-258bccd03ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.655896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.656167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.656295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.656559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] PATH_INFO: `/v3/projects/5934621267af4b838889fcbfb9febaea/users/26abb94392364bdc82f6bb508a611d44/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.675452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=5934621267af4b838889fcbfb9febaea, user_id=26abb94392364bdc82f6bb508a611d44, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.678892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.695193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.704997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 814/1611] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:07 2022] PUT /identity/v3/projects/5934621267af4b838889fcbfb9febaea/users/26abb94392364bdc82f6bb508a611d44/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:07.711424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:07.714471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7212fd36-0ee7-4708-8ce2-258bccd03ad5 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.719089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7212fd36-0ee7-4708-8ce2-258bccd03ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:07.764805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:07.812448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542cd10-0ada-43ce-bb11-2a34a8ffc0eb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:07.822150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:07.822945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.823221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.823490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.824179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:07.826154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c45d53c-3e39-4b1a-9d28-b9af3b94ab93 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:07.879898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 798/1612] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:07.881602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 815/1613] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/projects => generated 373 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:07.883048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.883351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.883653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.890831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa980d8-6537-4182-b215-520957afae71 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:07.891122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa980d8-6537-4182-b215-520957afae71 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:07.891382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa980d8-6537-4182-b215-520957afae71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:07.921638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] MFA Rules not processed for user `e7bca9820438452685bbc9db2ee90e02`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.923109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:07.950651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3fa980d8-6537-4182-b215-520957afae71 None None] MFA Rules not processed for user `2cf05c1f95264293ae851bb86ed6aeea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:07.952072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3fa980d8-6537-4182-b215-520957afae71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:08.058735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 816/1614] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:08.059912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.079710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 799/1615] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:08.081637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.082649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.083161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.133124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:08.134408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:08.152246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0614b6-2eaf-47f0-9dce-dff2deea600f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.162445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.167792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.168103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.168256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.265650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.296976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 800/1616] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:08.298060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.315714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f67252-9aa2-4ed4-8947-aa74b9af65ba None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.322232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.323281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.323576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.324293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.325715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.331298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.336945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.339327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.357085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 817/1617] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:08.358724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.382824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 801/1618] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:08 2022] POST /identity/v3/users => generated 293 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:08.382824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.398808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819bc23e-a0b1-4548-8a4a-03c88e76ef0d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.418589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.419224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.419518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.419773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.420584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.434482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e371868d-40b2-45df-a4e7-34ba713b2bc3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.513569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbccb5-b1fa-4024-a628-939aeece86b7 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.522944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.523396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.524366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.524366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.528257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 802/1619] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:08 2022] POST /identity/v3/projects => generated 391 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:08.547950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02dac-39e7-4795-a5ec-3fb27543b585 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.549825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02dac-39e7-4795-a5ec-3fb27543b585 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.552088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02dac-39e7-4795-a5ec-3fb27543b585 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.559219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 803/1620] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:35:08 2022] GET /identity => generated 270 bytes in 29 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:08.561560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d02dac-39e7-4795-a5ec-3fb27543b585 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.584601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.627623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.629403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.642744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 818/1621] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:08.643799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.669181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb910d68-d993-4220-add7-321e1116bf41 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.735224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d02dac-39e7-4795-a5ec-3fb27543b585 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.757416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.758211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.758575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.758780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.848874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.849981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.850434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.853312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.854988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.860992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.866493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:08.898460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 819/1622] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:08.902850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24668b-92eb-49c8-83bc-eef1112a3172 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 804/1623] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.997122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.997775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c745c7b-284e-406d-9466-09fcfda9aefa admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:08.997775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:08.997775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:08.997775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:08.998175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:08.998175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.000527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.039162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:09.044723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:09.091940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 820/1624] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:08 2022] POST /identity/v3/users => generated 302 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:09.093399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.108967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c9674e-c410-40db-8c7d-e95635311eb1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.123338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.125490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.128395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.128895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] PATH_INFO: `/v3/projects/8a67c38974c94779935c10af81cc11aa/users/16f6a23bf79f48e4a59ae91a55501e59/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.173005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a67c38974c94779935c10af81cc11aa, user_id=16f6a23bf79f48e4a59ae91a55501e59, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.182214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80930957-6b0c-4083-a8f3-72d997c914aa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.236398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 821/1625] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:09 2022] PUT /identity/v3/projects/8a67c38974c94779935c10af81cc11aa/users/16f6a23bf79f48e4a59ae91a55501e59/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:09.238386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012b9f8-3aa4-4c0d-a708-186b62cc2bc3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.238940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012b9f8-3aa4-4c0d-a708-186b62cc2bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.239166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012b9f8-3aa4-4c0d-a708-186b62cc2bc3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.251310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 822/1626] 10.222.0.16 () {54 vars in 912 bytes} [Tue Oct 4 08:35:09 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:09.253090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e012b9f8-3aa4-4c0d-a708-186b62cc2bc3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.321383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 805/1627] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:35:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 375 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:09.323246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.373338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e9669d-be04-49dd-ab6e-6f44b378a181 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.374071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e012b9f8-3aa4-4c0d-a708-186b62cc2bc3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.381546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.382754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.383206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.383684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.384571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.386781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.388379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a1fd54-bcfe-4dc3-bc42-1a15b940061d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.388760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.389158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.389983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.393138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.395383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.414315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 806/1628] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:09 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:09.421973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.422709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.423210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.428892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 823/1629] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:09 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:09.429880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.459501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:09.461972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:09.528703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:09.618976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 807/1630] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:35:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:09.621423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.642450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9487241-fe95-47ad-969e-3e360dc54971 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.651055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec900e0-53fc-44e8-9d37-3edc441e1af9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.661516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.662306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.662454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.662851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.663560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] PATH_INFO: `/v3/projects/8a67c38974c94779935c10af81cc11aa/users/16f6a23bf79f48e4a59ae91a55501e59/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.664050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.664425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.665018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.668606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a67c38974c94779935c10af81cc11aa, user_id=16f6a23bf79f48e4a59ae91a55501e59, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.676196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.721456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 808/1631] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:09 2022] PUT /identity/v3/projects/8a67c38974c94779935c10af81cc11aa/users/16f6a23bf79f48e4a59ae91a55501e59/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:09.722495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.814273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f612f5-990d-46ab-a4b0-92d1d42b0347 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:09.821074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:09.821770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.822432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.822774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] PATH_INFO: `/v3/projects/ba3bb90521cc481893702cebf4afafc9/users/902e98117a5b400681d87659a397c829/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:09.834690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:09.843525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba3bb90521cc481893702cebf4afafc9, user_id=902e98117a5b400681d87659a397c829, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.849355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8fb4e4-3bb7-445e-97a2-e099ffe0f73c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.852942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:09.854724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:09.882368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 824/1632] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:09.892695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:09.913532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 809/1633] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:09 2022] PUT /identity/v3/projects/ba3bb90521cc481893702cebf4afafc9/users/902e98117a5b400681d87659a397c829/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:09.916389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:09.916679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:09.916958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.028239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] MFA Rules not processed for user `16f6a23bf79f48e4a59ae91a55501e59`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:10.029524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:10.073259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc98d0-706d-4770-9547-7a22d7a83738 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:10.181621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 810/1634] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:09 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:10.182689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:10.218078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:10.222970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:10.224149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.225101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.225522 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.333818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8467933c-48f9-4c5b-9570-f93aeee282c6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:10.345735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:10.346280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.346627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.347056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.347718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:10.349315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:10.383795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 811/1635] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:10 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:10.384691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:10.401886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:10.403263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:10.411778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf3b81c-a76c-478a-9d8d-2d21c200d966 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:10.420788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:10.421255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.421255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.421546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.434507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 825/1636] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:10.435280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:10.442872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:10.444352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d996ce08-924a-4c17-8063-b655c56a3b8c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:10.463512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0a4f4-d6dd-4509-b45c-3fd9b14538f8 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:10.476249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 812/1637] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:10 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:10.489602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a866146-64a9-4c58-ae71-fdbe29f62912 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.493019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:10.493945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.494381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.494628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a866146-64a9-4c58-ae71-fdbe29f62912 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.495201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a866146-64a9-4c58-ae71-fdbe29f62912 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.495735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] PATH_INFO: `/v3/projects/ba3bb90521cc481893702cebf4afafc9/users/902e98117a5b400681d87659a397c829/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.503452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 813/1638] 10.222.0.16 () {54 vars in 912 bytes} [Tue Oct 4 08:35:10 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:10.505012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.506121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba3bb90521cc481893702cebf4afafc9, user_id=902e98117a5b400681d87659a397c829, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:10.506980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.508227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.511660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43073b7b-ff50-4f7e-83ec-dc5f5fbf9860 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:10.545173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 826/1639] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:10 2022] PUT /identity/v3/projects/ba3bb90521cc481893702cebf4afafc9/users/902e98117a5b400681d87659a397c829/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:10.551520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:10.553858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:10.556091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.556606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.557066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:10.619619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] MFA Rules not processed for user `902e98117a5b400681d87659a397c829`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:10.621349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:10.668084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:10.735187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 814/1640] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:35:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:10.736130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:10.798523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 827/1641] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:10 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:10.871815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d6024a-8ed7-4f85-871e-70c30534fcdf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:10.879230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:10.879865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:10.880142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:10.880721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.004569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:11.026394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.030478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.073623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 815/1642] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:10 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:11.514048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:11.530308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f01964-c0c7-4e8d-aea7-f992943e1876 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:11.550069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:11.550579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:11.550941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:11.551119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.586137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:11.589836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.591531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.598237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521068e9-4439-4e99-9ae0-9f09ee4b46e8 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:11.614040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:11.614507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:11.614811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:11.620192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 828/1643] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:11.620889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:11.649179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:11.651157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:11.674869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 816/1644] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:11 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:12.188997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:12.199084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c664a5b-11bf-4590-928d-de23d84c13fc service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:12.204570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:12.205434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:12.205763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:12.206034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:12.422399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:12.430856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cf81c-f099-45c7-b358-f5536044d595 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:12.473127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 829/1645] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:12 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:15.295659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:15.485936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6992648-7ffe-47c5-96e3-6433bd1043c7 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:15.493397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:15.494337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:15.494889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:15.495646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] PATH_INFO: `/v3/users/2cf05c1f95264293ae851bb86ed6aeea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:15.497194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cf05c1f95264293ae851bb86ed6aeea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:15.498768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:15.498768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Oct 04 08:35:15.498768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: Oct 04 08:35:15.500366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:15.712783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cf05c1f95264293ae851bb86ed6aeea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:15.764143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cf05c1f95264293ae851bb86ed6aeea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:15.836451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cf05c1f95264293ae851bb86ed6aeea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:15.837527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cf05c1f95264293ae851bb86ed6aeea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:15.838327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fd701086-1b8c-47c7-8f51-e79ca1321f67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cf05c1f95264293ae851bb86ed6aeea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:15.890487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 817/1646] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:15 2022] DELETE /identity/v3/users/2cf05c1f95264293ae851bb86ed6aeea => generated 0 bytes in 592 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:15.892091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:15.892596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:15.892667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:15.958637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:15.966633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:16.168163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 830/1647] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:16.225628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:16.226189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:16.226507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:16.227120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:16.228576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:16.270348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] MFA Rules not processed for user `87ca9fc2e013477cad3a3de8e8811274`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:16.273780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:16.301368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:16.324384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a161db28-f807-43d6-b4c2-3e785364de7e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:16.329580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:16.330283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:16.330651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:16.331039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:16.347436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:16.352420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:16.372212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 831/1648] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:16.396998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:16.435085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 818/1649] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:16 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:16.455363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:16.456823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:16.619196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93af12c-a24d-4af7-b445-80756cd98f14 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:16.631586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:16.633565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:16.634531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:16.635003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:16.660506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:16.669178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9377a3c8-569c-4f82-b696-3523e579d0b7 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:16.715035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 832/1650] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:16 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:17.464568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:17.466245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:17.554958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add03527-b670-4cff-b188-e2fa583998d0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:17.564585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:17.566178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:17.566832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:17.567359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] PATH_INFO: `/v3/projects/7f158aa9def14740b321dacf2fbacab1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:17.573321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f158aa9def14740b321dacf2fbacab1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:17.581118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:17.692439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f158aa9def14740b321dacf2fbacab1', 'request_id': 'req-0e78d169-0fa8-4bfb-815b-b3ea7710872e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:17.715024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f158aa9def14740b321dacf2fbacab1', 'request_id': 'req-0e78d169-0fa8-4bfb-815b-b3ea7710872e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:17.729287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f158aa9def14740b321dacf2fbacab1', 'request_id': 'req-0e78d169-0fa8-4bfb-815b-b3ea7710872e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:17.792782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] The token cache is being invalidate because project 7f158aa9def14740b321dacf2fbacab1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:17.793081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:17.811491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e78d169-0fa8-4bfb-815b-b3ea7710872e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:17.835872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:17.837379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:17.837864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:17.853348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 819/1651] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:17 2022] DELETE /identity/v3/projects/7f158aa9def14740b321dacf2fbacab1 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:17.894392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:17.897834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:18.188253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 833/1652] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:18.195454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.195738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.196012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.243017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:18.249052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:18.285678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 820/1653] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:18.291059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:18.307579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3182f7-08f1-4b87-8f39-838bdfef5130 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:18.323609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:18.324280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.329929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.330598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.332159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:18.334011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:18.341791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:18.382073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 834/1654] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:18.386159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:18.396832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3197e56b-b231-4e8b-be55-c457eedba659 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:18.414129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:18.414782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.415119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.415486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.416591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:18.426436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:18.501246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 821/1655] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:18 2022] POST /identity/v3/projects => generated 367 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:18.505312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:18.525058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b3f7a0-000c-4648-8043-2d6b51c0fd28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:18.763948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:18.772928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.773448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.773621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.774604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:18.780160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:18.794705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:18.835098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 835/1656] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:18 2022] POST /identity/v3/users => generated 284 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:18.839127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:18.862466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a377fd3d-5863-4642-b381-2b0e4495f52e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:18.875447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:18.878712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.880022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.882355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.885754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:18.888708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:18.914003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 822/1657] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:18 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:18.918757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:18.952241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e225c2-5fd0-4cb8-9616-5a0ef98be0d8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:18.966268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:18.968408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:18.968904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:18.969460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:18.970807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:18.973767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e69bff-ddec-499c-b14c-5ca621aaabae admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:18.990605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 836/1658] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:18 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:18.994404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:19.010892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c57438-1f9f-4186-acdd-850aef244184 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:19.023294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:19.024114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.025362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.025727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] PATH_INFO: `/v3/projects/b3739657364644c9a9d7f671bbd3aaba/users/e28311dd9f574f519d279f33ca7feb69/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.066174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3739657364644c9a9d7f671bbd3aaba, user_id=e28311dd9f574f519d279f33ca7feb69, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:19.070468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a70a81-bcfb-4990-a13a-61e3dd313ef0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:19.095556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 823/1659] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:18 2022] PUT /identity/v3/projects/b3739657364644c9a9d7f671bbd3aaba/users/e28311dd9f574f519d279f33ca7feb69/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:19.107688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.107998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.112268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.194566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] MFA Rules not processed for user `e28311dd9f574f519d279f33ca7feb69`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:19.196176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:19.199919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68311612-c52b-4f8c-af28-592e64ceff1b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.200740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68311612-c52b-4f8c-af28-592e64ceff1b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.201387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68311612-c52b-4f8c-af28-592e64ceff1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.284546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-68311612-c52b-4f8c-af28-592e64ceff1b None None] MFA Rules not processed for user `d002a844a71649d884b4ca870eb9d387`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:19.285806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-68311612-c52b-4f8c-af28-592e64ceff1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:19.332491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 837/1660] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:19.338057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:19.435399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:19.471190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 824/1661] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:19.482264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.482568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.483034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.531385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] MFA Rules not processed for user `d002a844a71649d884b4ca870eb9d387`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:19.532698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:19.558688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798b1d7-75b1-4359-b215-2416c11d542c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:19.560009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 825/1662] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:19.574276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:19.575819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:19.584730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:19.585368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.585650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.585922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.586623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:19.588670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:19.694272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 838/1663] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/projects => generated 369 bytes in 356 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:19.697799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:19.809422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aabbf0-517d-424b-ba73-b4ea3793d2fc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:19.900638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:19.941364 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203f3f9-7237-4eac-98e8-3d155483ea59 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:19.953558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:19.954442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:19.954882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:19.956028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:19.956825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:19.960830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.042022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 839/1664] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:19 2022] POST /identity/v3/users => generated 291 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:20.051713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:20.090426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db898ded-e4b4-4775-95c9-e2c5c350b85d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.109726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.113244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.118058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.118741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.122814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.127277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.127737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.127894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.128581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.131781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.218373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 840/1665] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:20 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:20.230691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:20.254062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222959b8-7734-436f-8bca-7493eb106ee0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.274422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.275162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.275393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.275697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] PATH_INFO: `/v3/projects/b5f1040f82f5446aae144aff4a53d880/users/5b80f8c9f1d548eaafc613343ba81062/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.336556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f1040f82f5446aae144aff4a53d880, user_id=5b80f8c9f1d548eaafc613343ba81062, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.349917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.362274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.365952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.386410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 841/1666] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:20 2022] PUT /identity/v3/projects/b5f1040f82f5446aae144aff4a53d880/users/5b80f8c9f1d548eaafc613343ba81062/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:20.390644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:20.397207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 826/1667] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:19 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 823 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:20.553735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82a231-a1cc-49b2-96e0-bf2441a6fc8d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.576247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.577332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.577332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.577332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.577976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.580049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3822f54-54a5-4e9a-aca4-8bb5a385800d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.602708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 842/1668] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:20 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:20.606803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:20.626576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f48443-4fab-48bd-8e38-a9d02084667e service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:20.650048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:20.655690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.656047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.656409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] PATH_INFO: `/v3/projects/b5f1040f82f5446aae144aff4a53d880/users/5b80f8c9f1d548eaafc613343ba81062/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.665432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f1040f82f5446aae144aff4a53d880, user_id=5b80f8c9f1d548eaafc613343ba81062, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:20.673929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:20.708111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 827/1669] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:20 2022] PUT /identity/v3/projects/b5f1040f82f5446aae144aff4a53d880/users/5b80f8c9f1d548eaafc613343ba81062/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:20.718339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:20.718711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:20.718930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:20.798503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] MFA Rules not processed for user `5b80f8c9f1d548eaafc613343ba81062`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:20.799802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:20.988247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 843/1670] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:20 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:21.001519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:21.004170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:21.026005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:21.138004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3ce4f6-d044-4aa9-b340-c2c12bae9f83 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:21.148226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:21.149646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:21.149646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:21.149646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:21.166920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:21.168475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624c5ab-08f4-4916-a9d1-1dad9a238359 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:21.181867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 828/1671] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:21.683937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:21.686108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:21.744617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98552d50-0669-4a81-9b3d-7d40e3016948 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:21.745434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98552d50-0669-4a81-9b3d-7d40e3016948 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:21.745848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98552d50-0669-4a81-9b3d-7d40e3016948 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:21.747810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 829/1672] 10.222.0.16 () {54 vars in 912 bytes} [Tue Oct 4 08:35:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 04 08:35:21.798615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:21.799238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:21.799710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:21.826063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:21.927258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:21.935353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.000609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf41632d-d387-43b7-abcf-fc91c74af057 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:22.029589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:22.030250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.030531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.030829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] PATH_INFO: `/v3/users/e7bca9820438452685bbc9db2ee90e02` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.036002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7bca9820438452685bbc9db2ee90e02)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:22.053606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:22.223290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 830/1673] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:35:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:22.246100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7bca9820438452685bbc9db2ee90e02'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.268728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:22.270444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.285255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7bca9820438452685bbc9db2ee90e02'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.290053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c322a-cb8d-4067-8457-9efc01f7a63c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:22.309337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:22.309969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.310240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.310505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.351719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7bca9820438452685bbc9db2ee90e02'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.352585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7bca9820438452685bbc9db2ee90e02'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.354021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:22.363514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:22.379067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ebf8461-30a2-4b8a-a7e6-e8ecce29d644 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7bca9820438452685bbc9db2ee90e02'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.401956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 831/1674] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:22 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:22.403003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:22.403723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 844/1675] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:21 2022] DELETE /identity/v3/users/e7bca9820438452685bbc9db2ee90e02 => generated 0 bytes in 720 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:22.405318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.410100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.410438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.411873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.477076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e50c09-fbb2-42f7-9202-534e321ca341 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:22.503720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:22.511427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.672699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:22.675639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.676228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.677392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] PATH_INFO: `/v3/users/baf5ab01e28944258aa2fdfe7951b504` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.680824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baf5ab01e28944258aa2fdfe7951b504)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:22.697235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:22.697471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 845/1676] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:22.762104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:22.763952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.784993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:22.812091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baf5ab01e28944258aa2fdfe7951b504'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.839872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baf5ab01e28944258aa2fdfe7951b504'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.879036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:22.888556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdee1148-f7ae-4c0a-a806-c4b71a3e5696 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:22.889081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baf5ab01e28944258aa2fdfe7951b504'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.894338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baf5ab01e28944258aa2fdfe7951b504'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.896252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-674d8917-1178-425c-9772-3db0bbfd9e45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baf5ab01e28944258aa2fdfe7951b504'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:22.898803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:22.899493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.900471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.900916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.905802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:22.938528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:22.941032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 832/1677] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:22 2022] DELETE /identity/v3/users/baf5ab01e28944258aa2fdfe7951b504 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:22.941857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:22.947540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc63b0b-b027-43bf-a65e-0ab5b24721a5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:22.948160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc63b0b-b027-43bf-a65e-0ab5b24721a5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:22.949274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc63b0b-b027-43bf-a65e-0ab5b24721a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:22.969251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 846/1678] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:23.013378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fcc63b0b-b027-43bf-a65e-0ab5b24721a5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:23.016973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fcc63b0b-b027-43bf-a65e-0ab5b24721a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:23.265934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 833/1679] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:23.335396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:23.336856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:23.361988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49e3b-fb19-411c-bfa5-4b12b617f7e6 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:23.521513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:23.522289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:23.522608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:23.523093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:23.556727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:23.560231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c55caf-9934-4a35-b070-14d322b87fb6 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:23.599085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 847/1680] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:24.064048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:24.064579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:24.064807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:24.113703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] MFA Rules not processed for user `87ca9fc2e013477cad3a3de8e8811274`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:24.116310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:24.362784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 834/1681] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:24 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:24.929256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:24.929769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:24.930012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:24.973975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:24.976485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.049775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] MFA Rules not processed for user `0c1de2efe2ed42508d7ab4a2e9d91e92`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:25.054129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.123069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:25.135260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc097ed-30f3-4853-85a0-0ced3d671da1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:25.146340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0481e952-a582-497b-88c9-f414927579db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:25.147432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481e952-a582-497b-88c9-f414927579db service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:25.148144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481e952-a582-497b-88c9-f414927579db service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:25.148821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481e952-a582-497b-88c9-f414927579db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:25.151947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0481e952-a582-497b-88c9-f414927579db service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.172439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0481e952-a582-497b-88c9-f414927579db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:25.175307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0481e952-a582-497b-88c9-f414927579db service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:25.191908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 835/1682] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:25.193166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0481e952-a582-497b-88c9-f414927579db service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:25.195905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0481e952-a582-497b-88c9-f414927579db service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.214245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0481e952-a582-497b-88c9-f414927579db service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:25.238082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:25.239451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:25.240028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:25.240980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] PATH_INFO: `/v3/users/e28311dd9f574f519d279f33ca7feb69` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:25.244695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e28311dd9f574f519d279f33ca7feb69)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:25.248102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:25.303821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 848/1683] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:25.402057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e28311dd9f574f519d279f33ca7feb69'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:25.405812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:25.407558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.415934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e28311dd9f574f519d279f33ca7feb69'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:25.464728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e28311dd9f574f519d279f33ca7feb69'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:25.466268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e28311dd9f574f519d279f33ca7feb69'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:25.467016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15effe0e-bc12-4cde-ab9b-47cfee9b82f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e28311dd9f574f519d279f33ca7feb69'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:25.497744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 836/1684] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:25 2022] DELETE /identity/v3/users/e28311dd9f574f519d279f33ca7feb69 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:25.507939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:25.509044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:25.509362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:25.568487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:25.574838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.594116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12afc8f1-515b-4927-8f9a-05cca7afc8fd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:25.649751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:25.826714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:25.827951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:25.828421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:25.829089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:25.837066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.910682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 837/1685] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:25.914026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:25.916715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:25.988360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da7ced5-a384-4bd9-a422-44901227fa0f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:26.023775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:26.024751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.025214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.025682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] PATH_INFO: `/v3/projects/d61d65375ccf4967b7545338ce72bf68` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.044718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d61d65375ccf4967b7545338ce72bf68)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:26.072785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:26.187852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:26.199025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:26.233525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 849/1686] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 827 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:26.237052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:26.244237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:26.279192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd61d65375ccf4967b7545338ce72bf68', 'request_id': 'req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.303110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd61d65375ccf4967b7545338ce72bf68', 'request_id': 'req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.311617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:26.324313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd61d65375ccf4967b7545338ce72bf68', 'request_id': 'req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.377993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] The token cache is being invalidate because project d61d65375ccf4967b7545338ce72bf68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:26.378184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.379060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c9acd1de-f0d6-4099-a36b-75c9b35dd17c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.382484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 838/1687] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:25 2022] DELETE /identity/v3/projects/d61d65375ccf4967b7545338ce72bf68 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:26.384198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.386220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.386649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.388609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 839/1688] 10.222.0.16 () {54 vars in 912 bytes} [Tue Oct 4 08:35:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:26.390172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:26.391994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:26.513074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446197f7-13f0-44a3-9862-ecbc44167e9d service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:26.670370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec68777-0f77-45a0-adaa-ae55ce798722 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:26.685477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:26.686320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.686617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.686891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] PATH_INFO: `/v3/projects/33adc74b068441a1b62341362286fe83` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.704284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33adc74b068441a1b62341362286fe83)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:26.715691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:26.764786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:26.765530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.767131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.768841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.774010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:26.837200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:26.841617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d7ad9c-fb81-4928-81e0-e4c50f97f02b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:26.886237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 850/1689] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 651 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:26.891016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:26.892983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:26.894652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:26.957378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33adc74b068441a1b62341362286fe83', 'request_id': 'req-6a549318-a011-40e7-851d-25df291d0134'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:26.967487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:26.971019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:26.988427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33adc74b068441a1b62341362286fe83', 'request_id': 'req-6a549318-a011-40e7-851d-25df291d0134'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.014751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:27.017425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33adc74b068441a1b62341362286fe83', 'request_id': 'req-6a549318-a011-40e7-851d-25df291d0134'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.048478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] The token cache is being invalidate because project 33adc74b068441a1b62341362286fe83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:27.048793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.050962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6a549318-a011-40e7-851d-25df291d0134 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.057704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 840/1690] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:26 2022] DELETE /identity/v3/projects/33adc74b068441a1b62341362286fe83 => generated 0 bytes in 669 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:27.067511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b568b86-9e27-4528-86cf-605fd808140f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.067832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b568b86-9e27-4528-86cf-605fd808140f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.068153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b568b86-9e27-4528-86cf-605fd808140f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.129557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2b568b86-9e27-4528-86cf-605fd808140f None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:27.144965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2b568b86-9e27-4528-86cf-605fd808140f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.265327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 851/1691] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:27.275649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:27.279239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.317530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 841/1692] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:35:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:27.321200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.322523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.322934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.340734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6663c19-1a47-40db-8cdf-b347c237c7fc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.383964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.388420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.388735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.390144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] PATH_INFO: `/v3/users/902e98117a5b400681d87659a397c829` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.394419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:27.395350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.398027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=902e98117a5b400681d87659a397c829)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.406049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:27.456395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 842/1693] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:27.456395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:27.459879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.512839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36e0df5-1294-4f04-9b06-7128acec3853 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.531941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.532619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.533468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.533737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.576635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.579713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '902e98117a5b400681d87659a397c829'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.580817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34ce63b-fadc-42a2-ac85-824f69ed8caf service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:27.596414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '902e98117a5b400681d87659a397c829'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.608434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 843/1694] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:27.611983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ab9eb-36c2-4ef4-b68c-e1d68889aaea None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.613784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ab9eb-36c2-4ef4-b68c-e1d68889aaea None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.614084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993ab9eb-36c2-4ef4-b68c-e1d68889aaea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.635748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:27.657346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '902e98117a5b400681d87659a397c829'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.660953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '902e98117a5b400681d87659a397c829'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.697831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '902e98117a5b400681d87659a397c829'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:27.706639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 852/1695] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:27 2022] DELETE /identity/v3/users/902e98117a5b400681d87659a397c829 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:27.711395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:27.712704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.743789 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-993ab9eb-36c2-4ef4-b68c-e1d68889aaea None None] MFA Rules not processed for user `26abb94392364bdc82f6bb508a611d44`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:27.758136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-993ab9eb-36c2-4ef4-b68c-e1d68889aaea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:27.936588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542fce7-1659-439e-bac5-b6d1dc7fea7f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:27.955288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:27.955930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:27.956209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:27.956482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:27.957150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:27.957452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:27.963651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.011391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 853/1696] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:28.016322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 844/1697] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:27 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:28.030892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.032540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.032828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.033201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.053116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab6a2bc-d16e-4834-a5a4-abcb99199ffd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.064357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.064518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.065034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.065310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.066144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.072834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.078924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] MFA Rules not processed for user `26abb94392364bdc82f6bb508a611d44`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:28.080355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:28.090031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 845/1698] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:28 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:28.153297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 854/1699] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:28 2022] POST /identity/v3/projects => generated 407 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:28.163473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.200579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47fdfe5-0595-4177-93e9-aaf2f83312eb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.318226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.321450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:28.329957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:28.330866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.334272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.335387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.335810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.336682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.339765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.380483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 846/1700] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:28 2022] POST /identity/v3/users => generated 304 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:28.381283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.395564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8706547f-6ea1-4484-95c6-97451fd92c16 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.403191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.403928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.404474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.404802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.405836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.407752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.443914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 847/1701] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:28 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:28.445080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.474941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae57462-c8ec-4029-be43-df47deada85a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.519303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe273b44-7bad-4f9b-9048-e2b99e50b000 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.541932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.546394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.548310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.549175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.554578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.554901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.555566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.555845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.556230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.560838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.600308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 848/1702] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:28 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:28.602064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.629753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:28.649660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738151ae-2fb6-414f-a555-d9bf4eeda95f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.655648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:28.656527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:28.657645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:28.658242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] PATH_INFO: `/v3/projects/b3739657364644c9a9d7f671bbd3aaba` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:28.659669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3739657364644c9a9d7f671bbd3aaba)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.661960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.803599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3739657364644c9a9d7f671bbd3aaba', 'request_id': 'req-e0417c77-758e-40aa-a30e-e528ce522b42'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:28.808550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:28.818970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:28.823276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:28.835645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3739657364644c9a9d7f671bbd3aaba', 'request_id': 'req-e0417c77-758e-40aa-a30e-e528ce522b42'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:28.870476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3739657364644c9a9d7f671bbd3aaba', 'request_id': 'req-e0417c77-758e-40aa-a30e-e528ce522b42'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:28.878862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 855/1703] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:28 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:28.883623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.905612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23d148-2d72-4855-b45b-423b355c3422 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:28.926394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] The token cache is being invalidate because project b3739657364644c9a9d7f671bbd3aaba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:28.929582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:28.934518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:28.942001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 849/1704] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:28 2022] DELETE /identity/v3/projects/b3739657364644c9a9d7f671bbd3aaba => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:28.947917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:28.950379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:29.225256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:29.231357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.231643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.231920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] PATH_INFO: `/v3/projects/422d021bffee4409a4f52ef36a85bab5/users/a71c5c0370694adbbadfd03b400dcbe6/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.308822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0417c77-758e-40aa-a30e-e528ce522b42 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:29.319413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] RBAC: Authorizing `identity:create_grant(project_id=422d021bffee4409a4f52ef36a85bab5, user_id=a71c5c0370694adbbadfd03b400dcbe6, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:29.323709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d160af91-9dec-4793-a524-d16f51af1775 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:29.328730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:29.330431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.331477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.332023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] PATH_INFO: `/v3/users/5b80f8c9f1d548eaafc613343ba81062` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.341813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b80f8c9f1d548eaafc613343ba81062)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:29.355409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:29.411386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 856/1705] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:28 2022] PUT /identity/v3/projects/422d021bffee4409a4f52ef36a85bab5/users/a71c5c0370694adbbadfd03b400dcbe6/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:29.414827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.415206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.415539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.511993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b80f8c9f1d548eaafc613343ba81062'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:29.537454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b80f8c9f1d548eaafc613343ba81062'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:29.568318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b80f8c9f1d548eaafc613343ba81062'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:29.571482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] MFA Rules not processed for user `a71c5c0370694adbbadfd03b400dcbe6`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:29.572739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:29.573391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b80f8c9f1d548eaafc613343ba81062'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:29.577004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f5513c0f-a068-46a2-93fa-c4ee1cbf48a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b80f8c9f1d548eaafc613343ba81062'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:29.604807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:29.605912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 850/1706] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:28 2022] DELETE /identity/v3/users/5b80f8c9f1d548eaafc613343ba81062 => generated 0 bytes in 658 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:29.735486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:29.735962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:29.736317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:29.758900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 857/1707] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:29 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:29.764150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:29.768908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:29.885736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:29.888786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:29.945436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2450dd1f-06ad-4886-b384-50eb2ef93607 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.033663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.084897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.088243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.088701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.089074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.090100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.093424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.168176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 858/1708] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:29 2022] POST /identity/v3/projects => generated 407 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:30.172193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.176863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:30.249793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 851/1709] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:35:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 515 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:35:30.261766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.312608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879291f3-f199-40b8-8820-0de01816fae6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.363513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:30.418674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb25bd-fcd7-45b0-9bfb-85a14aaa8caf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.425002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.425802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.426191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.426625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] PATH_INFO: `/v3/projects/ba3bb90521cc481893702cebf4afafc9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.429055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba3bb90521cc481893702cebf4afafc9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.430282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 04 08:35:30.430282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Oct 04 08:35:30.430282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: Oct 04 08:35:30.431796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.562465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba3bb90521cc481893702cebf4afafc9', 'request_id': 'req-e9db6224-3092-4e1e-9073-979263158106'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:30.572069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.574506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.574858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.577128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.596814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.597252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba3bb90521cc481893702cebf4afafc9', 'request_id': 'req-e9db6224-3092-4e1e-9073-979263158106'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:30.618148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7df0b4-610f-404b-8811-ae05ae19285f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.620137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba3bb90521cc481893702cebf4afafc9', 'request_id': 'req-e9db6224-3092-4e1e-9073-979263158106'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:30.657664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] The token cache is being invalidate because project ba3bb90521cc481893702cebf4afafc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:30.660431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:30.667224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:30.669854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 859/1710] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:30 2022] DELETE /identity/v3/projects/ba3bb90521cc481893702cebf4afafc9 => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:30.673019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.676833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:30.705851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 852/1711] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:30 2022] POST /identity/v3/users => generated 310 bytes in 450 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:30.708524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.708687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.717425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.764692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:30.769183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:30.933028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 853/1712] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:30.943261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:30.950786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:30.973029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a44ed58-bf1b-49c9-b101-22ebd6674389 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:30.985774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:30.986729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:30.987119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:30.987515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:30.988305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:30.990960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba22db9-de22-4350-b2fb-83c97e685037 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:30.991196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9db6224-3092-4e1e-9073-979263158106 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.000151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.000793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.001067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.001337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.015426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:31.015870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 854/1713] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:31.017950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.018425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.018893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.075811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:31.077649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:31.095810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 855/1714] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:31.098101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.145432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e231aec-3312-4449-8e28-1b0ffdc6b33f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.167421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.168083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.168360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.168627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] PATH_INFO: `/v3/projects/04d16cf69b1345f384f28f2772a65cdf/users/25ff6ca70199440e8fca4b5489febc99/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.253604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04d16cf69b1345f384f28f2772a65cdf, user_id=25ff6ca70199440e8fca4b5489febc99, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.261040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3801a2-e2b1-4d2c-b407-1141230c3990 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.269880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:31.274378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.279738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.288976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 856/1715] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:31 2022] PUT /identity/v3/projects/04d16cf69b1345f384f28f2772a65cdf/users/25ff6ca70199440e8fca4b5489febc99/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:31.293390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.293723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.294012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.305895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 860/1716] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:35:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 634 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:35:31.309314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.348557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] MFA Rules not processed for user `0c1de2efe2ed42508d7ab4a2e9d91e92`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:31.354107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:31.355747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24d9cd-cb9a-46ec-826a-026ec8a3d709 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.410666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:31.608994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.609806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.610095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.610583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.611500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:31.611819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.615668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.646500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 857/1717] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:31.656904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.660759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 861/1718] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:31.666181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.701199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f72abf-74c7-425f-824a-9180336e1488 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.711806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319395f9-8f35-4ea9-8e58-896ed524895a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.733826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.734222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.734502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.734801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.735449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.741376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.743425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.745187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.745702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.746640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.750925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.761069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.839019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 858/1719] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:31 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:31.853197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.887981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6b725-493d-455e-ad2b-9baf33b13cab admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:31.892944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 862/1720] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:31 2022] POST /identity/v3/projects => generated 367 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:31.897573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:31.901778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:31.902476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.902778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.908518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] PATH_INFO: `/v3/projects/04d16cf69b1345f384f28f2772a65cdf/users/25ff6ca70199440e8fca4b5489febc99/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:31.915795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] RBAC: Authorizing `identity:create_grant(project_id=04d16cf69b1345f384f28f2772a65cdf, user_id=25ff6ca70199440e8fca4b5489febc99, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:31.926479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed20317-64cd-4933-b628-8fd16ffd055a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:31.994096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 859/1721] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:31 2022] PUT /identity/v3/projects/04d16cf69b1345f384f28f2772a65cdf/users/25ff6ca70199440e8fca4b5489febc99/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:31.999611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:31.999611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:31.999611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.098527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] MFA Rules not processed for user `25ff6ca70199440e8fca4b5489febc99`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:32.099958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:32.157414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec8b29f-b4e8-41d9-abcd-991f7bb166d2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:32.368941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 860/1722] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:31 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:32.378311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:32.379031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.379326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.379662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.380459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:32.384402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:32.384805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:32.387041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:32.477574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 863/1723] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:31 2022] POST /identity/v3/users => generated 284 bytes in 579 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:32.479707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:32.504299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:32.514012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0306ea-acf8-495c-9596-a0c55206360e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:32.526865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:32.527594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.527940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.528312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.529069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:32.530814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:32.551118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b10af9-776c-43cb-bcfb-3ce14f542148 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:32.555375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:32.555994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.556267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.556715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.559341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 864/1724] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:32 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:32.562881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:32.582276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a6a65-f7c3-4005-a134-cd1e1100b23d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:32.584660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:32.587972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:32.615931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:32.616570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.616760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.617032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.617744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:32.629280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 861/1725] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:32 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:32.632051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d09adf8-2c76-4101-b64f-7791fc7627e9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:32.649192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 865/1726] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:32 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:32.653877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:32.682907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b6f27-9a54-429f-8fad-42eaed8ae8b9 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:32.723631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:32.724539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.724961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.725294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] PATH_INFO: `/v3/projects/360fec4c951f4f529cba143c92f134c0/users/09af3129349c4c7d95f8c909f311cfcf/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.782377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] RBAC: Authorizing `identity:create_grant(project_id=360fec4c951f4f529cba143c92f134c0, user_id=09af3129349c4c7d95f8c909f311cfcf, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:32.793138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:32.860839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 862/1727] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:32 2022] PUT /identity/v3/projects/360fec4c951f4f529cba143c92f134c0/users/09af3129349c4c7d95f8c909f311cfcf/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:32.871984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:32.872574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:32.873034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:32.962987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] MFA Rules not processed for user `09af3129349c4c7d95f8c909f311cfcf`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:32.964887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:33.134158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:33.141419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:33.258259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 866/1728] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:32 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:33.259233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:33.265138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:33.351086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b48fd8-9fef-4198-8ef6-586352d13791 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:33.370300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:33.370974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:33.371258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:33.371525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:33.377749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:33.519994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56d18cf-fa0d-4023-a820-78fae606111d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:33.527222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:33.528155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:33.528576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:33.528986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] PATH_INFO: `/v3/projects/b5f1040f82f5446aae144aff4a53d880` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:33.530537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5f1040f82f5446aae144aff4a53d880)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:33.552836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:33.596618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:33.599527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:33.620573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 863/1729] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:33 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:33.622104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:33.699862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2218cf-da06-4bee-a4af-fb6a33a0ce65 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:33.763162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5f1040f82f5446aae144aff4a53d880', 'request_id': 'req-6df0ad3b-8dd6-4869-9a28-714e978d8349'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:33.792312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5f1040f82f5446aae144aff4a53d880', 'request_id': 'req-6df0ad3b-8dd6-4869-9a28-714e978d8349'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:33.824211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5f1040f82f5446aae144aff4a53d880', 'request_id': 'req-6df0ad3b-8dd6-4869-9a28-714e978d8349'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:33.867603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] The token cache is being invalidate because project b5f1040f82f5446aae144aff4a53d880 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:33.867956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:33.873971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6df0ad3b-8dd6-4869-9a28-714e978d8349 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:33.887869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 867/1730] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:33 2022] DELETE /identity/v3/projects/b5f1040f82f5446aae144aff4a53d880 => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:33.890813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5c553-c66e-46e4-a641-c728ba74e973 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:33.891064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5c553-c66e-46e4-a641-c728ba74e973 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:33.891298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf5c553-c66e-46e4-a641-c728ba74e973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:33.916022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:33.974617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2bf5c553-c66e-46e4-a641-c728ba74e973 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:33.976774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:33.977525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2bf5c553-c66e-46e4-a641-c728ba74e973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:33.981963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:33.982316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:33.982625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:33.983301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:33.997431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.012110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 868/1731] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:34.033993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.037194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.040753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.075118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 864/1732] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:33 2022] POST /identity/v3/projects => generated 367 bytes in 453 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:34.076253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.084192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:34.109378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:34.112727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:34.245611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:34.307363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b447d6-5b8b-4368-bd65-98d6365d1e87 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:34.322922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:34.323564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.323842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.324188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.324994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:34.327286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.412791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 865/1733] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:34 2022] POST /identity/v3/users => generated 290 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:34.413694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.423646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 869/1734] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:34.426838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.470452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c318f6a-9556-401d-99dd-0cce6fe679d7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:34.485255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ffb6e9-a766-406f-94dc-1fb8af0fb41e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:34.523056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:34.523890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.528074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.528365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.529076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:34.546863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:34.548725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.549010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.549291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.554183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:34.554355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.554500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:34.556172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.622716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 866/1735] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:34 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:34.622927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 870/1736] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:34.624276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.626821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.654425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444e5fb-4e08-4da9-b319-ac7dda18bc3e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:34.667041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:34.667669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.667952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.669393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.669621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:34.676711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.718432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4d2833-df70-40f2-b551-b6743a715fe2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:34.739300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:34.742194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:34.742874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:34.743230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] PATH_INFO: `/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1/users/038ba1ac13234c5cbdfa8db15f6f43ca/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:34.788231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 867/1737] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:34 2022] POST /identity/v3/projects => generated 379 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:34.790899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:34.830781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b60a3d4a17254a5aa3fde8b224ab9ad1, user_id=038ba1ac13234c5cbdfa8db15f6f43ca, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:34.847001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:34.891030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 871/1738] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:34 2022] PUT /identity/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1/users/038ba1ac13234c5cbdfa8db15f6f43ca/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:34.908041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:35.108568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cf1ecd-cbcd-4ca0-9f5f-10dbdbe7b91e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:35.205224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf608727-872e-4288-9808-03f41d5e39f1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:35.223120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:35.225885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.226863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.227181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.227868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:35.243291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:35.299091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 872/1739] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:34 2022] GET /identity/v3/roles => generated 1420 bytes in 396 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:35.305722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:35.345573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5a6a4c-1b4e-4879-821f-ce6f9a4e64ff admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:35.356924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:35.357613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.357903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.359276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.359969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:35.361481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:35.367055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.368002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.368914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] PATH_INFO: `/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1/users/038ba1ac13234c5cbdfa8db15f6f43ca/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.371925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b0b28-517e-48a2-aaa1-2f301d3480e9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:35.375977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b60a3d4a17254a5aa3fde8b224ab9ad1, user_id=038ba1ac13234c5cbdfa8db15f6f43ca, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:35.381326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eed2448-1e15-44fd-a631-8954244d5413 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:35.429279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 873/1740] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:35 2022] PUT /identity/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1/users/038ba1ac13234c5cbdfa8db15f6f43ca/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:35.433354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.433658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.434342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.449078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 868/1741] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:34 2022] POST /identity/v3/users => generated 290 bytes in 660 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:35.451209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.451209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.451209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.507227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:35.513837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:35.554676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] MFA Rules not processed for user `038ba1ac13234c5cbdfa8db15f6f43ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:35.557429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:35.617281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:35.659121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:35.722689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 874/1742] 10.222.0.17 () {58 vars in 1015 bytes} [Tue Oct 4 08:35:35 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:35.724856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:35.782846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 869/1743] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:35 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:35.803517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:35.805231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:35.911666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6209615-3e68-4853-9d65-5e93ea7242c9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:35.916992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:35.917619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:35.917897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:35.918168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:35.918996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:35.920634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:35.938673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 875/1744] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:35 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:35.938673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:35.945560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:36.004377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9855e-ee31-42ed-9d0f-78da5495d7c1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.021796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.026590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.028518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.028518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.065296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:36.071359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:36.074284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0bc8f-1201-4933-b64e-18153ee79ee6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.090253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.092574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.093729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.094797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.104509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 876/1745] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:36.108311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:36.133719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:36.139238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:36.175590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cbb827-45b2-4cc2-bdef-1ecaee14d324 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.189107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.189921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.190252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 870/1746] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:35 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:36.190407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.190929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.191811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:36.194282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e21c2-a56e-4fb3-bf41-999320dc4f62 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:36.206892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 877/1747] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:36 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:36.211614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:36.226830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9509b89-2bba-4e39-895a-bf377a0c090f service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.232854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.233667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.234011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.234391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] PATH_INFO: `/v3/projects/2c6319f8616142829541317b6b7157dc/users/ae72bf24bb3145bb90b2e6628317f666/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.260020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c6319f8616142829541317b6b7157dc, user_id=ae72bf24bb3145bb90b2e6628317f666, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:36.274884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:36.311983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 871/1748] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:36 2022] PUT /identity/v3/projects/2c6319f8616142829541317b6b7157dc/users/ae72bf24bb3145bb90b2e6628317f666/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:35:36.324967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.329516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.336515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.442377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] MFA Rules not processed for user `ae72bf24bb3145bb90b2e6628317f666`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:36.446011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:36.638497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:36.645594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:36.714035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 878/1749] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:36.724089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:36.844499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec16ccd-284c-464f-9746-d8ec6b6bacb2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.853938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.858808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.859159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.859515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.880027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:36.880801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:36.928210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4e2925-df79-43ae-b1a7-4d32bc3da3e9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:36.951345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:36.952379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:36.952692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:36.952952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:36.953654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:36.962832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.054793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 879/1750] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:36 2022] POST /identity/v3/projects => generated 381 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:37.058903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:37.085627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:37.096490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.101819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.128040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 872/1751] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:36 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:37.198594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7659ad-4218-43ce-b6a1-3365a5032f76 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:37.206868 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:37.207560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.209524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.209992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.210810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.217289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.294295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 880/1752] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:37 2022] POST /identity/v3/users => generated 297 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:37.296037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:37.312543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47bec52-d91e-484b-896b-7971f063121f service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:37.322079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:37.323302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.324027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.324687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.325703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.330673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.366491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 873/1753] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:35:37.372207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:37.388815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7d87ea-10d4-40c5-8a88-2ae9f15e6166 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:37.405532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:37.406906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.408894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.409266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] PATH_INFO: `/v3/projects/5276ae3602ce4aa7bf642484b3c69fde/users/df99942c96094935b341f70f0d8d64fe/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.451749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5276ae3602ce4aa7bf642484b3c69fde, user_id=df99942c96094935b341f70f0d8d64fe, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.457875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.486481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 881/1754] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:37 2022] PUT /identity/v3/projects/5276ae3602ce4aa7bf642484b3c69fde/users/df99942c96094935b341f70f0d8d64fe/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:35:37.492063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:37.513014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa8953-32d4-4d4e-a89f-7e30e391b344 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:37.706974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:37.707702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.708211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.709271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.710147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.712159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f842275d-cb30-4e50-877d-64a3f822ae60 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.765469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 874/1755] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:37.769898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:37.792809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e6ac42-a02e-4c78-8a3b-012d5cced9e6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:37.818529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:37.825654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.828130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.828479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] PATH_INFO: `/v3/projects/5276ae3602ce4aa7bf642484b3c69fde/users/df99942c96094935b341f70f0d8d64fe/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.846653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5276ae3602ce4aa7bf642484b3c69fde, user_id=df99942c96094935b341f70f0d8d64fe, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:37.850144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:37.940558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a998a5-4ba8-46b1-8bf8-37187dd74541 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:37.941103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a998a5-4ba8-46b1-8bf8-37187dd74541 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:37.941103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a998a5-4ba8-46b1-8bf8-37187dd74541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:37.946463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 882/1756] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:37 2022] PUT /identity/v3/projects/5276ae3602ce4aa7bf642484b3c69fde/users/df99942c96094935b341f70f0d8d64fe/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:38.026920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b9a998a5-4ba8-46b1-8bf8-37187dd74541 None None] MFA Rules not processed for user `df99942c96094935b341f70f0d8d64fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:38.030973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b9a998a5-4ba8-46b1-8bf8-37187dd74541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:38.215401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 875/1757] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:37 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:38.225328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:38.314006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef86a90-215f-4dfa-bbd6-d15762655b1f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:38.320325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:38.321071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:38.321427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:38.321767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:38.333670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:38.335512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:38.348325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 883/1758] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:38 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:38.418006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:38.418803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:38.419296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:38.474387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] MFA Rules not processed for user `26abb94392364bdc82f6bb508a611d44`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:38.476886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:38.569479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 876/1759] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:38 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:39.117240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:39.129031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705f3e9b-8351-4a9d-94bf-10e62676eb4a service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:39.159836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df99942c96094935b341f70f0d8d64fe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5276ae3602ce4aa7bf642484b3c69fde', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:39.174944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:39.175358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:39.175745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:39.179197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:39.179630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:39.182292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:39.209833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 884/1760] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:39.209993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:39.222947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e26686-6252-4bb8-a99f-7e24f50ce9fe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:39.228924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df99942c96094935b341f70f0d8d64fe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5276ae3602ce4aa7bf642484b3c69fde', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:39.229488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:39.229773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:39.230050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:39.232555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:39.234513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:39.247660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:39.290255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 877/1761] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:39 2022] POST /identity/v3/projects => generated 354 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:39.294672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:39.310591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8222f-7a06-43f2-8a10-cf1cb1d92c3c tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:39.363163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:39.470444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df99942c96094935b341f70f0d8d64fe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5276ae3602ce4aa7bf642484b3c69fde', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:39.471947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:39.472879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:39.473409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:39.475121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:39.477096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:39.479974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a23d4b-6fd6-4f30-ac0c-e17e5e173923 tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:39.520202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 885/1762] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:39.641775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:39.683930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ade1f-e5cd-4f97-9c96-2b3336339c6b tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:39.695300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df99942c96094935b341f70f0d8d64fe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5276ae3602ce4aa7bf642484b3c69fde', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:39.696046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:39.696340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:39.696613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] PATH_INFO: `/v3/projects/9680d66b1b5e4226a1326f6043c2799d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:39.707507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorizing `identity:delete_project(project_id=9680d66b1b5e4226a1326f6043c2799d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:39.709933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:39.895501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9680d66b1b5e4226a1326f6043c2799d', 'request_id': 'req-af20afb8-7a87-4dd2-b048-436e7bd697de'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:39.918694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9680d66b1b5e4226a1326f6043c2799d', 'request_id': 'req-af20afb8-7a87-4dd2-b048-436e7bd697de'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:39.940295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9680d66b1b5e4226a1326f6043c2799d', 'request_id': 'req-af20afb8-7a87-4dd2-b048-436e7bd697de'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:39.973036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] The token cache is being invalidate because project 9680d66b1b5e4226a1326f6043c2799d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:39.979154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:39.980210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-af20afb8-7a87-4dd2-b048-436e7bd697de tempest-VolumeQuotaClassesTest-1703238184 tempest-VolumeQuotaClassesTest-1703238184-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:39.984903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 878/1763] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:39 2022] DELETE /identity/v3/projects/9680d66b1b5e4226a1326f6043c2799d => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:40.435974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:40.436523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:40.436523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:40.437865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 886/1764] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:35:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:40.472752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9174788a-79c7-4003-ade4-53612f440428 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:40.473761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9174788a-79c7-4003-ade4-53612f440428 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:40.474271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9174788a-79c7-4003-ade4-53612f440428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:40.535866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9174788a-79c7-4003-ade4-53612f440428 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:40.537701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9174788a-79c7-4003-ade4-53612f440428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:40.855445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 879/1765] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:35:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:41.393770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:41.396969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:41.791468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ceeb60-b572-4b2c-a49c-131c1a88d0df None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:41.822409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:41.823120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:41.823429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:41.823714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] PATH_INFO: `/v3/users/ae72bf24bb3145bb90b2e6628317f666` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:41.831153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae72bf24bb3145bb90b2e6628317f666)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:41.854689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:42.060890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae72bf24bb3145bb90b2e6628317f666'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:42.084871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae72bf24bb3145bb90b2e6628317f666'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:42.129864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae72bf24bb3145bb90b2e6628317f666'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:42.131794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae72bf24bb3145bb90b2e6628317f666'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:42.189760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae72bf24bb3145bb90b2e6628317f666'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:42.217777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 887/1766] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:41 2022] DELETE /identity/v3/users/ae72bf24bb3145bb90b2e6628317f666 => generated 0 bytes in 825 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:42.222278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:42.223281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:42.224418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:42.273382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:42.277140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:42.468302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 880/1767] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:42.581401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:42.584156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:42.671661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:42.759199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d9110-67c9-4a5a-bd9a-4b15eb492412 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:42.774369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:42.777044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:42.777524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:42.779688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:42.813028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:42.815341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:42.844212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 888/1768] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:43.581036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:43.582805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:43.610149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:43.825919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d04f5e-f3cd-49ed-a84d-a78f31577ca9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:43.864527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:43.866469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:43.867822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:43.869407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:43.875516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:43.912646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:43.915505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:43.938372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 881/1769] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:43.986738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:43.988580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:44.022353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e892ed8-4df3-4eb2-9246-916da9a1462c service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:44.041685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:44.042517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:44.042861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:44.043235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] PATH_INFO: `/v3/projects/2c6319f8616142829541317b6b7157dc` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:44.044576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c6319f8616142829541317b6b7157dc)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:44.047587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:44.138738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c6319f8616142829541317b6b7157dc', 'request_id': 'req-a39ed0fa-4228-46bc-9226-610a07b6e187'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.159794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c6319f8616142829541317b6b7157dc', 'request_id': 'req-a39ed0fa-4228-46bc-9226-610a07b6e187'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.172849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c6319f8616142829541317b6b7157dc', 'request_id': 'req-a39ed0fa-4228-46bc-9226-610a07b6e187'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.187731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] The token cache is being invalidate because project 2c6319f8616142829541317b6b7157dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:44.188155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.188858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.190959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 889/1770] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:43 2022] DELETE /identity/v3/projects/2c6319f8616142829541317b6b7157dc => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:44.194941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:44.225383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8ef30-c253-4d46-ae25-f4ea799ac4ae service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:44.294383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:44.295177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:44.295677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:44.296004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] PATH_INFO: `/v3/users/df99942c96094935b341f70f0d8d64fe` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:44.297512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df99942c96094935b341f70f0d8d64fe)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:44.299804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:44.356067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df99942c96094935b341f70f0d8d64fe'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.365494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df99942c96094935b341f70f0d8d64fe'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.378620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df99942c96094935b341f70f0d8d64fe'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.379440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df99942c96094935b341f70f0d8d64fe'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.380159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9ecb99e4-7d51-47fd-8b8e-8aa1d9b73765 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df99942c96094935b341f70f0d8d64fe'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.394755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 882/1771] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:44 2022] DELETE /identity/v3/users/df99942c96094935b341f70f0d8d64fe => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:44.751658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:44.753265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:44.766295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39ed0fa-4228-46bc-9226-610a07b6e187 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:44.857376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:44.858054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:44.858331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:44.858633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] PATH_INFO: `/v3/projects/5276ae3602ce4aa7bf642484b3c69fde` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:44.859931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5276ae3602ce4aa7bf642484b3c69fde)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:44.861866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:44.927540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5276ae3602ce4aa7bf642484b3c69fde', 'request_id': 'req-4af903ff-9e17-48eb-bddd-584530d2b641'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.933988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5276ae3602ce4aa7bf642484b3c69fde', 'request_id': 'req-4af903ff-9e17-48eb-bddd-584530d2b641'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.941155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5276ae3602ce4aa7bf642484b3c69fde', 'request_id': 'req-4af903ff-9e17-48eb-bddd-584530d2b641'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.943315 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:44.956103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] The token cache is being invalidate because project 5276ae3602ce4aa7bf642484b3c69fde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:44.956392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.957042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4af903ff-9e17-48eb-bddd-584530d2b641 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:44.962352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 890/1772] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:44 2022] DELETE /identity/v3/projects/5276ae3602ce4aa7bf642484b3c69fde => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:44.965040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:44.965636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:44.965768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:44.988974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:44.999563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:45.001702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:45.074057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 883/1773] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:45.083609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.084351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.084351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.104611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:45.105824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:45.113986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 891/1774] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:45.119063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.129300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282aa541-671b-4473-9c0b-91098c5c8fc4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.136152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.139066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.139344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.139614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.140281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:45.140581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.142201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.156081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 884/1775] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:35:45.161721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.172520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50203a9-869b-46b5-aa2b-cf4a31b76db8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.177331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.177949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.178229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.178504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.179242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.181199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.212459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 892/1776] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:45.216678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.228785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41938d41-2795-44d9-8888-f9bbfdf955e8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.294423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.295071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.295345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.295611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.296248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.298187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.324885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 885/1777] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:45.326434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.338399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b053106-7dbf-4512-8f28-f5bb43f78507 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.343299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.343958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.344236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.344507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.345171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.346834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.359169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 893/1778] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:45.363091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.376844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61193ee8-3975-457a-abaf-ffa5659451a3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.382206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.382856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.383138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.383407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.384082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.385681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead295b1-07d7-4d62-a9c8-7663dc5bbc3a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.397702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 886/1779] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:45.402098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.412092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017d278e-9b5c-44e2-b666-e5e1e04c168b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.416773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.417434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.417708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.417976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] PATH_INFO: `/v3/projects/ff968fce8939452880a6350ca7218e96/users/0ca744f3dd61477b9fe67e4355e28c76/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.433564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff968fce8939452880a6350ca7218e96, user_id=0ca744f3dd61477b9fe67e4355e28c76, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.436245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.451345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 894/1780] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:45 2022] PUT /identity/v3/projects/ff968fce8939452880a6350ca7218e96/users/0ca744f3dd61477b9fe67e4355e28c76/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:35:45.464307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.464509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.464781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.494456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] MFA Rules not processed for user `0ca744f3dd61477b9fe67e4355e28c76`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:45.495749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:45.566436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 887/1781] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:45.572101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.657280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ada8a09-46a5-422c-9404-a99d2d06dbf5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.665271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.666016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.666362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.666739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.667473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.669501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.699730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 895/1782] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/projects => generated 385 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:45.703907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.710156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:45.718768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701255d9-b486-4bf2-9cdb-22637f667c6e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.787845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.788491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.788767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.789175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.789833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.791787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.823254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 888/1783] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:45 2022] POST /identity/v3/users => generated 299 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:45.828001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.857260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb5f6-1366-4648-9bdd-21bda990d51a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.862258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.862913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.863199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.863473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.864180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.865992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.878986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 896/1784] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:45 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:45.883599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:45.894203 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed530bb-c1da-4a33-911e-61a653b888cc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:45.899743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:45.900455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:45.900841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:45.901171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] PATH_INFO: `/v3/projects/2728f037c73f4876a79b6480b9e1ebd9/users/5ff8c5da68aa40ce8e21e12917f723f7/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:45.940272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2728f037c73f4876a79b6480b9e1ebd9, user_id=5ff8c5da68aa40ce8e21e12917f723f7, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:45.946729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:45.986598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 889/1785] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:45 2022] PUT /identity/v3/projects/2728f037c73f4876a79b6480b9e1ebd9/users/5ff8c5da68aa40ce8e21e12917f723f7/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:35:45.990678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:46.022004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aa195-f5a1-4136-9d57-ed76e769c20e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:46.161056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:46.162943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:46.163412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:46.164065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:46.165005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:46.167483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b56b15-caf1-4923-b4ab-5c1e1083618d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:46.171296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:46.186389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:46.193963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 897/1786] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:45 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:46.201566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a14d20b-d081-4970-9459-c18aec8de307 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:46.218625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:46.218625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:46.218625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:46.218625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] PATH_INFO: `/v3/projects/2728f037c73f4876a79b6480b9e1ebd9/users/5ff8c5da68aa40ce8e21e12917f723f7/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:46.222647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2728f037c73f4876a79b6480b9e1ebd9, user_id=5ff8c5da68aa40ce8e21e12917f723f7, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:46.225368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:46.256166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 890/1787] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:46 2022] PUT /identity/v3/projects/2728f037c73f4876a79b6480b9e1ebd9/users/5ff8c5da68aa40ce8e21e12917f723f7/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:46.257111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:46.257400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:46.257702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:46.296255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] MFA Rules not processed for user `5ff8c5da68aa40ce8e21e12917f723f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:46.297879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:46.379481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 898/1788] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:46 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:46.391761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:46.393698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:46.457099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5b35f2-6f2a-463a-b965-81d519461fc5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:46.461443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:46.462258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:46.462717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:46.463171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:46.534816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:46.536264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:46.546605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 891/1789] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:49.867391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:49.869399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.013642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5a1fea-64d1-48ee-a0ca-e6c13453eed6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:50.018065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:50.018762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.019037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.019439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] PATH_INFO: `/v3/users/a71c5c0370694adbbadfd03b400dcbe6` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.034720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a71c5c0370694adbbadfd03b400dcbe6)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:50.040010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:50.138928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:50.142806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a71c5c0370694adbbadfd03b400dcbe6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.148216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.176152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a71c5c0370694adbbadfd03b400dcbe6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.196808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a71c5c0370694adbbadfd03b400dcbe6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.197899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a71c5c0370694adbbadfd03b400dcbe6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.199901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997ef20c-a735-4506-b9f4-c29669f63f95 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:50.209111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0b6705-4b78-4a87-8cc8-ce897a17280e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a71c5c0370694adbbadfd03b400dcbe6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.210336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:50.210336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.210336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.210661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] PATH_INFO: `/v3/users/09af3129349c4c7d95f8c909f311cfcf` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.213529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:50.213899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=09af3129349c4c7d95f8c909f311cfcf)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:50.219948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 899/1790] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:49 2022] DELETE /identity/v3/users/a71c5c0370694adbbadfd03b400dcbe6 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:50.221348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:50.225115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ccc6b6-061b-4af9-a177-781cb042430c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.226984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ccc6b6-061b-4af9-a177-781cb042430c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.227267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ccc6b6-061b-4af9-a177-781cb042430c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.229350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:50.231005 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 900/1791] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:35:50 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:50.232419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866c577-33cd-4bfc-bf93-fe1341c54dc2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.232815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866c577-33cd-4bfc-bf93-fe1341c54dc2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.233116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866c577-33cd-4bfc-bf93-fe1341c54dc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.282272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5866c577-33cd-4bfc-bf93-fe1341c54dc2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:50.283552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5866c577-33cd-4bfc-bf93-fe1341c54dc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.353492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09af3129349c4c7d95f8c909f311cfcf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.386373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 901/1792] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:50.386642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09af3129349c4c7d95f8c909f311cfcf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.386759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.386759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.386759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.402163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09af3129349c4c7d95f8c909f311cfcf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.403047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09af3129349c4c7d95f8c909f311cfcf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.403760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-135b2e6c-a05d-4f37-bb2e-756e645e10e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09af3129349c4c7d95f8c909f311cfcf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:50.422815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 892/1793] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:50 2022] DELETE /identity/v3/users/09af3129349c4c7d95f8c909f311cfcf => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:35:50.427432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.427818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.428317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.463780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:50.464375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:50.465576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.467743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.554900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 893/1794] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:50.555899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:50.559193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.640072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 902/1795] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:35:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:50.640973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:50.654748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.672267 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d08cd7-1822-4eed-bd10-8719b94c8ede None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:50.677009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:50.678263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.678794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.679297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.681016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb49194-608b-4a53-b053-dc9c43c86063 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:50.682129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.684858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:50.685473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.685749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.686077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.704745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:50.706345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:50.707902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:50.709920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:50.720890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 894/1796] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:50.722134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:50.724727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:50.736069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 903/1797] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:50.753331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc11c782-3a77-428a-819c-c0e5f9389510 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:50.761397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:50.762192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:50.762598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:50.763178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:50.780171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:50.787747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:50.826650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 895/1798] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:51.403039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:51.405014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:51.424696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f8696e-43a1-4771-82f0-5d09a104443e service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:51.431215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:51.431793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:51.432104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:51.432374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] PATH_INFO: `/v3/projects/422d021bffee4409a4f52ef36a85bab5` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:51.433805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=422d021bffee4409a4f52ef36a85bab5)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:51.436922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:51.519891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '422d021bffee4409a4f52ef36a85bab5', 'request_id': 'req-7a27c125-8a78-4906-9d0d-c36c709443f7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.528813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '422d021bffee4409a4f52ef36a85bab5', 'request_id': 'req-7a27c125-8a78-4906-9d0d-c36c709443f7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.538603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '422d021bffee4409a4f52ef36a85bab5', 'request_id': 'req-7a27c125-8a78-4906-9d0d-c36c709443f7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.557417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] The token cache is being invalidate because project 422d021bffee4409a4f52ef36a85bab5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:51.557834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.558886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.561181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 904/1799] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:51 2022] DELETE /identity/v3/projects/422d021bffee4409a4f52ef36a85bab5 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:51.565494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:51.578631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef38a-55e4-4fa6-be0b-5550aec276e0 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:51.589490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:51.653442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:51.654116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:51.654390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:51.654703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] PATH_INFO: `/v3/users/25ff6ca70199440e8fca4b5489febc99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:51.656447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=25ff6ca70199440e8fca4b5489febc99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:51.659543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:51.725892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25ff6ca70199440e8fca4b5489febc99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.735894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25ff6ca70199440e8fca4b5489febc99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.751261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25ff6ca70199440e8fca4b5489febc99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.752003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25ff6ca70199440e8fca4b5489febc99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.752616 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25ff6ca70199440e8fca4b5489febc99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:51.768296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 896/1800] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:51 2022] DELETE /identity/v3/users/25ff6ca70199440e8fca4b5489febc99 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:52.245564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:52.248897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:52.269713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:52.282447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a27c125-8a78-4906-9d0d-c36c709443f7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:52.454179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:52.456235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:52.456992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:52.459226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] PATH_INFO: `/v3/projects/360fec4c951f4f529cba143c92f134c0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:52.477395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] RBAC: Authorizing `identity:delete_project(project_id=360fec4c951f4f529cba143c92f134c0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:52.485776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:52.505310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:52.508034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:52.552068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e96217-441d-4d0c-b0da-d34c3dcada8a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:52.588431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:52.593496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:52.594639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:52.595477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] PATH_INFO: `/v3/projects/04d16cf69b1345f384f28f2772a65cdf` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:52.599331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] RBAC: Authorizing `identity:delete_project(project_id=04d16cf69b1345f384f28f2772a65cdf)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:52.605348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:52.618057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '360fec4c951f4f529cba143c92f134c0', 'request_id': 'req-12aadd3c-3040-44ef-b3ff-6b8c316eb784'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.633879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '360fec4c951f4f529cba143c92f134c0', 'request_id': 'req-12aadd3c-3040-44ef-b3ff-6b8c316eb784'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.651081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '360fec4c951f4f529cba143c92f134c0', 'request_id': 'req-12aadd3c-3040-44ef-b3ff-6b8c316eb784'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.696906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] The token cache is being invalidate because project 360fec4c951f4f529cba143c92f134c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:52.697151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.699365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.703857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 905/1801] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:52 2022] DELETE /identity/v3/projects/360fec4c951f4f529cba143c92f134c0 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:52.709959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:52.712332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:52.791377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04d16cf69b1345f384f28f2772a65cdf', 'request_id': 'req-9c541636-b5f1-484a-b8e9-5ef0f082d47b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.813931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04d16cf69b1345f384f28f2772a65cdf', 'request_id': 'req-9c541636-b5f1-484a-b8e9-5ef0f082d47b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.831974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04d16cf69b1345f384f28f2772a65cdf', 'request_id': 'req-9c541636-b5f1-484a-b8e9-5ef0f082d47b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.856844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] The token cache is being invalidate because project 04d16cf69b1345f384f28f2772a65cdf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:52.857139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.859931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c541636-b5f1-484a-b8e9-5ef0f082d47b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:52.863520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 897/1802] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:52 2022] DELETE /identity/v3/projects/04d16cf69b1345f384f28f2772a65cdf => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:52.867188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aadd3c-3040-44ef-b3ff-6b8c316eb784 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:52.871338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b4dd9-f9b7-4160-8522-d59fb3360e8d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:52.871623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b4dd9-f9b7-4160-8522-d59fb3360e8d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:52.871895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b4dd9-f9b7-4160-8522-d59fb3360e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:52.911929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a94b4dd9-f9b7-4160-8522-d59fb3360e8d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:52.916352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a94b4dd9-f9b7-4160-8522-d59fb3360e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:52.939166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:52.964849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:53.001294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.005785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.005785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.005785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] PATH_INFO: `/v3/users/038ba1ac13234c5cbdfa8db15f6f43ca` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.005785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=038ba1ac13234c5cbdfa8db15f6f43ca)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.007960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.061563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 898/1803] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:53.072828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.073048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.073350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.096601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '038ba1ac13234c5cbdfa8db15f6f43ca'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:53.099271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:53.100639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:53.113615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '038ba1ac13234c5cbdfa8db15f6f43ca'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:53.124009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 899/1804] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:53.124009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.139936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '038ba1ac13234c5cbdfa8db15f6f43ca'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:53.146927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '038ba1ac13234c5cbdfa8db15f6f43ca'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:53.147078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b1090a-965b-4fd8-88e3-8db6694cab31 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.152488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.153412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.153583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.154433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.155520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:53.155864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.157969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.160335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '038ba1ac13234c5cbdfa8db15f6f43ca'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:53.173538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 906/1805] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:52 2022] DELETE /identity/v3/users/038ba1ac13234c5cbdfa8db15f6f43ca => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:53.189845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 900/1806] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:35:53.196040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.203070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:53.382637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e766d07-cc4b-4c0e-83d3-946e04e0bb70 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.409498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.410066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.410583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.413491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.413491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.418623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.477404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.477895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 907/1807] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:53 2022] POST /identity/v3/projects => generated 375 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:53.530479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eeb3c2-327a-4f86-8d2f-434346eb60db admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.717987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.718782 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.719137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.719906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.720819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.722847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.765223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 901/1808] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:53 2022] POST /identity/v3/users => generated 288 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:35:53.766176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.780581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817c3f9-779b-4426-b7fd-ecd0c293253f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.790418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.791115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.791192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.791455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.792123 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.793776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.807770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 908/1809] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:53 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:53.812008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.827449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec74d570-0ec5-43c5-b17b-bebb9a32154e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.842707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.843380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.843659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.844208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.845037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.847271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.881480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 902/1810] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:53 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:35:53.881795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.899157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a343cf-cc54-4adc-9f1b-4a85ceecb701 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:53.909267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:53.909947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:53.910218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:53.910485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] PATH_INFO: `/v3/projects/a9da6747c5ea46108ebfb64439adb7ad/users/3a9914f7c6d845829962007696b30455/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:53.949250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9da6747c5ea46108ebfb64439adb7ad, user_id=3a9914f7c6d845829962007696b30455, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:53.958289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2061c3b-1778-4185-ba67-a1af7073a27b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:53.988738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:53.991867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:53.997161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 909/1811] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:53 2022] PUT /identity/v3/projects/a9da6747c5ea46108ebfb64439adb7ad/users/3a9914f7c6d845829962007696b30455/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:54.011057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:54.012082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:54.012574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:54.072514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:54.095997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] MFA Rules not processed for user `3a9914f7c6d845829962007696b30455`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:54.104859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:54.181525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24fd147-556c-4bea-b00c-a00fdb42f2a7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:54.249058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:54.325417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:54.326287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:54.326457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:54.328750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:54.339265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:54.383447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 910/1812] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:54.487574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:54.489196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:54.533824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 903/1813] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:35:54.541871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:54.544258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:54.592901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2201170-fdbe-4f47-9931-0e77f0ce71ba None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:54.615318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:54.616040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:54.616496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:54.616709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:54.646737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:54.648641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafa0f5-6136-426a-898b-eaebcb045eb4 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:54.673511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 911/1814] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:54 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:55.728025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:55.731026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:55.820040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a183e02-11a7-40d2-8fbc-0bea0c5ba14a service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:55.841595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:55.842148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:55.842519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:55.842851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] PATH_INFO: `/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:55.844614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=b60a3d4a17254a5aa3fde8b224ab9ad1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:55.846888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:55.916248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b60a3d4a17254a5aa3fde8b224ab9ad1', 'request_id': 'req-386118ec-39cd-47a4-a166-5a71465b77dd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:55.924284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b60a3d4a17254a5aa3fde8b224ab9ad1', 'request_id': 'req-386118ec-39cd-47a4-a166-5a71465b77dd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:55.935998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b60a3d4a17254a5aa3fde8b224ab9ad1', 'request_id': 'req-386118ec-39cd-47a4-a166-5a71465b77dd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:55.953163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] The token cache is being invalidate because project b60a3d4a17254a5aa3fde8b224ab9ad1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:35:55.953464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:55.954245 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-386118ec-39cd-47a4-a166-5a71465b77dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:35:55.956143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 904/1815] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:35:55 2022] DELETE /identity/v3/projects/b60a3d4a17254a5aa3fde8b224ab9ad1 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:55.963080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:55.963975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:55.964256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:55.992894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:55.994486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:56.114859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 912/1816] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:56.122685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.122972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.123242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.156239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:56.157395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:56.173337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 905/1817] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:56.174079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.185800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca27ad56-c78d-45dc-a45b-2c7db32c0b47 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.210244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.210244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.210244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.210244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.210854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:35:56.210854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.210854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.215636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 913/1818] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:35:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:35:56.220481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.225693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:56.234163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b296967-f35c-4efe-8c1f-1cb9f92a4bd6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.247178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.248275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.249554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.249554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.250638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.252997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.290728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 906/1819] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:35:56 2022] POST /identity/v3/projects => generated 383 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:56.291926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.303430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f3acd1-8351-4359-820c-7ba54b302f9a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.365697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.366613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.366887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.367337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.367984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.370004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.395653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 914/1820] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:35:56 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:35:56.399936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.411502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1989d-2d38-40fe-a371-9440388682eb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.416507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.417120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.417398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.417666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.418322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.419950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.435602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 907/1821] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:35:56.439657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.446270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:35:56.454715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4c4ea-07c5-4056-8df3-8a71afcb041d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.464301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.464946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.465213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.466136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.466136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.467794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dd52b-6ca4-49cb-b383-3c3ff3def163 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.486803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 915/1822] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:35:56 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:35:56.487084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.498027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac4db3-94c6-47df-95f0-c3d7ac007914 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.503952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.504668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.504948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.505223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] PATH_INFO: `/v3/projects/a00e35a1b97b49f6b503517fcfba3a3d/users/5c835f8463d2466cad9943bf78b8a662/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.524376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] RBAC: Authorizing `identity:create_grant(project_id=a00e35a1b97b49f6b503517fcfba3a3d, user_id=5c835f8463d2466cad9943bf78b8a662, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.526743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.552085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 908/1823] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:35:56 2022] PUT /identity/v3/projects/a00e35a1b97b49f6b503517fcfba3a3d/users/5c835f8463d2466cad9943bf78b8a662/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:35:56.553831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdf243-ffd0-4cfd-a1c3-e964fa83972a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.554117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdf243-ffd0-4cfd-a1c3-e964fa83972a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.554519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fdf243-ffd0-4cfd-a1c3-e964fa83972a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.596386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b2fdf243-ffd0-4cfd-a1c3-e964fa83972a None None] MFA Rules not processed for user `5c835f8463d2466cad9943bf78b8a662`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:56.597983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2fdf243-ffd0-4cfd-a1c3-e964fa83972a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:56.728396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 916/1824] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:35:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:56.746020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:56.748895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:56.841865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111185e6-6693-48c4-b293-05b13f7bf5de admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:35:56.845751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:35:56.846485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:56.846867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:56.847081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:56.858328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:35:56.860481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d8cdbd-b8d8-49ff-8dfb-1c4ab534e41e service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:35:56.872229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 909/1825] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:35:56 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:35:58.877581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60410348-a48e-4962-9ad5-322646373afb None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:58.878238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60410348-a48e-4962-9ad5-322646373afb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:58.878238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60410348-a48e-4962-9ad5-322646373afb None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:58.881158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 917/1826] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:35:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:35:58.920365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:35:58.922423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:35:58.924289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:35:58.984662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:35:58.987920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:35:59.226775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 910/1827] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:35:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:35:59.815164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60410348-a48e-4962-9ad5-322646373afb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:35:59.819767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-60410348-a48e-4962-9ad5-322646373afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:00.014087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60410348-a48e-4962-9ad5-322646373afb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:00.020947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:00.021686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.022033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.022375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] PATH_INFO: `/v3/users/3a9914f7c6d845829962007696b30455` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:00.023726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a9914f7c6d845829962007696b30455)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:00.026092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:00.028211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:00.030132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:00.066840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0797e4-3162-4ed9-bd8b-5b92541a4420 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:00.094536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:00.095286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.095638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.096010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:00.105617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a9914f7c6d845829962007696b30455'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.111647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:00.113292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8ae4c4-a8ce-49d7-a44f-3402d96a4a9f service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:00.123170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a9914f7c6d845829962007696b30455'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.136196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 911/1828] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:36:00.140950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a9914f7c6d845829962007696b30455'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.141765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a9914f7c6d845829962007696b30455'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.152069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a9914f7c6d845829962007696b30455'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.160023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:00.169064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 918/1829] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:35:59 2022] DELETE /identity/v3/users/3a9914f7c6d845829962007696b30455 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:00.173908 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.174224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.174795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:00.198381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:00.199903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:00.228619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:00.284184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 912/1830] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:00.297628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:00.299498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:00.382330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f59455-5a89-4c88-b432-8be6db6af521 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:00.386356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:00.387022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.387299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.387650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:00.401213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:00.402949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a72bb6-07c1-489b-9850-cafd63542046 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:00.414972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 919/1831] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:00.814460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:00.816181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:00.834108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ee122b-4dd6-4e40-a3cb-998067c13785 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:00.839922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:00.840654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.840931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.841379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] PATH_INFO: `/v3/projects/a9da6747c5ea46108ebfb64439adb7ad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:00.842859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9da6747c5ea46108ebfb64439adb7ad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:00.844958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:00.919648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9da6747c5ea46108ebfb64439adb7ad', 'request_id': 'req-ab59dc6c-014c-4e4a-bd61-a51f70d47608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.926944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9da6747c5ea46108ebfb64439adb7ad', 'request_id': 'req-ab59dc6c-014c-4e4a-bd61-a51f70d47608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.938593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9da6747c5ea46108ebfb64439adb7ad', 'request_id': 'req-ab59dc6c-014c-4e4a-bd61-a51f70d47608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.952423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] The token cache is being invalidate because project a9da6747c5ea46108ebfb64439adb7ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:00.952689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.953374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ab59dc6c-014c-4e4a-bd61-a51f70d47608 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:00.954920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 913/1832] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:00 2022] DELETE /identity/v3/projects/a9da6747c5ea46108ebfb64439adb7ad => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:00.963621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:00.963997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:00.964828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.002626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:01.003900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:01.080713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 920/1833] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:01.090106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.090579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.090984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.117199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:01.118430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:01.127969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 914/1834] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:01.133043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.143458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca72458-11f5-473b-ae49-e912d384bb8d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.149342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.150271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.150758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.151234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.152146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:01.152791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.154838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.170607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 921/1835] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:01.174927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.185669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fda8f5-5fdd-489d-bbd8-8a095dd194a3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.192731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.193615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.193615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.193894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.194579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.197499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.234855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 915/1836] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/projects => generated 385 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:01.236459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.247885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a1baaf-4c69-4df3-98d7-65ebd6757f9b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.315917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.316543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.316921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.317762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.318408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.320439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.344006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 922/1837] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:01.348710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.359536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e9589c-f9d8-4a6a-8ae4-3982cb9240a7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.364979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.365611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.365893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.366170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.367043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.368697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.381531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 916/1838] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:01.385808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.397205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb5707-c442-4599-8458-b5c5696c187d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.402920 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.403667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.404020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.404365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.405091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.406821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f8d43e-3d79-4331-8b77-fb73f3da3116 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.421394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 923/1839] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:01.425686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.437001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d854738-7077-4b0a-a9c5-a1feeff24f07 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.444521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.445311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.445664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.446379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] PATH_INFO: `/v3/projects/d9e60959d3794ad1b9a88e15893a453a/users/e6fe45d17ef64333822b847bc51c1d24/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.465831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9e60959d3794ad1b9a88e15893a453a, user_id=e6fe45d17ef64333822b847bc51c1d24, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.468572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.483294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 917/1840] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:01 2022] PUT /identity/v3/projects/d9e60959d3794ad1b9a88e15893a453a/users/e6fe45d17ef64333822b847bc51c1d24/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:01.495599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.495873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.496945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.516916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:01.527829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] MFA Rules not processed for user `e6fe45d17ef64333822b847bc51c1d24`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:01.528950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:01.598517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 924/1841] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:01.603415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.661951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:01.685021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfcadfc-0b1d-44fa-8176-4970b0ee42ce admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.690600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.691210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.691489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.691776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.692416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.694342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.737831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 918/1842] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/projects => generated 383 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:01.743571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.755458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacac531-5800-40af-9b5d-e99ca8681197 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.835403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.836142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.836484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.836821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.837505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.839485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.864863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 925/1843] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/users => generated 292 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:01.866321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.876793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad170639-2e35-4145-9d5d-a650981b50be admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.882401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.883041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.883311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.883592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.884215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.885794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.898763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 919/1844] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:36:01.903841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.915676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7f1826-f56e-46b7-a301-ecc2ef7025b8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.920643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.921255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.921534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.921806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.922450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.924076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c1ab88-559d-4a3a-b380-5444a2ee6fb7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.934478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 926/1845] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:01.938655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:01.949200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ecf61a-b7cb-4475-a88d-9d553853a117 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:01.954314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:01.954989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.955306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.955582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] PATH_INFO: `/v3/projects/cbda626d8c314a21b56d943b78d13f94/users/4166dfecb3a547e79050d4ef29720551/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:01.970621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbda626d8c314a21b56d943b78d13f94, user_id=4166dfecb3a547e79050d4ef29720551, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:01.973354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:01.986101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 920/1846] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:01 2022] PUT /identity/v3/projects/cbda626d8c314a21b56d943b78d13f94/users/4166dfecb3a547e79050d4ef29720551/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:01.996483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:01.996770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:01.997043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.026356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] MFA Rules not processed for user `4166dfecb3a547e79050d4ef29720551`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:02.027602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:02.092098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 927/1847] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:01 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:02.096707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.166969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e0b587-c70f-49b3-8c04-c02a82a6e581 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.172015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.172636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.172912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.173180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.173827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.177785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.209279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 921/1848] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:02 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:02.210431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.222238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b1e77-f7dd-4bae-80de-41d234e3e9e9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.226466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:02.294588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.294588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.294588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.295193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.295424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.299186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.331816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 928/1849] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:02 2022] POST /identity/v3/users => generated 299 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:02.334339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.345861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7580414c-9ac4-475e-94e8-767a3652e5e7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.351985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.352597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.352868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.353131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.353787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.355421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.366589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 922/1850] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:02.370732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.381282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b5332-f0f0-4513-a506-f433403d5aa4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.386200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.386941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.387235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.387613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] PATH_INFO: `/v3/projects/541fdcf6bac045f2ac69d9e46307d590/users/ae9ef579ec204a6abdf7f28ef00e3f9c/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.404098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] RBAC: Authorizing `identity:create_grant(project_id=541fdcf6bac045f2ac69d9e46307d590, user_id=ae9ef579ec204a6abdf7f28ef00e3f9c, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.406808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.421334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 929/1851] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:02 2022] PUT /identity/v3/projects/541fdcf6bac045f2ac69d9e46307d590/users/ae9ef579ec204a6abdf7f28ef00e3f9c/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:02.425398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.437284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5b749f-767c-4923-9bf2-d2c897bf806e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.502157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.502786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.503063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.503330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.503963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.506177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378cd1fd-6052-41fb-a16b-d49fcec4e342 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.516176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 923/1852] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:02 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:02.521121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.531651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf81993c-0863-4a4c-b6c2-4539ae6e285b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.536786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.537465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.537743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.538014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] PATH_INFO: `/v3/projects/541fdcf6bac045f2ac69d9e46307d590/users/ae9ef579ec204a6abdf7f28ef00e3f9c/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.539932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] RBAC: Authorizing `identity:create_grant(project_id=541fdcf6bac045f2ac69d9e46307d590, user_id=ae9ef579ec204a6abdf7f28ef00e3f9c, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.542577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:02.553063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 930/1853] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:02 2022] PUT /identity/v3/projects/541fdcf6bac045f2ac69d9e46307d590/users/ae9ef579ec204a6abdf7f28ef00e3f9c/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:02.563344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.563642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.563860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.578752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:02.623101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] MFA Rules not processed for user `ae9ef579ec204a6abdf7f28ef00e3f9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:02.625036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:02.761611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 924/1854] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:02.776038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:02.778348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:02.868842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022a20e-8fb6-4a0b-934a-54381d051ebe admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:02.877484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:02.877762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:02.878076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:02.878415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:02.996731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:02.999042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:03.019803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 931/1855] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:03.829306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:03.831362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:03.917131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa196e36-c209-4547-a413-f0bfc61f1298 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:03.922401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:03.923173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:03.923523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:03.923791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] PATH_INFO: `/v3/users/0ca744f3dd61477b9fe67e4355e28c76` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:03.924943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ca744f3dd61477b9fe67e4355e28c76)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:03.927855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:03.953296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:03.954925 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:03.972279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4f17b-df18-4cc3-a19f-6bddddb61f46 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:03.978396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:03.979611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:03.979953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:03.980244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] PATH_INFO: `/v3/users/d9e8091bc8824c29be0a285767a3d32d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:03.981683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ca744f3dd61477b9fe67e4355e28c76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:03.981815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9e8091bc8824c29be0a285767a3d32d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:03.984294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:03.989067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ca744f3dd61477b9fe67e4355e28c76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:03.990332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:04.002793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ca744f3dd61477b9fe67e4355e28c76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.003538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ca744f3dd61477b9fe67e4355e28c76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.004153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-150781a9-fd99-4917-a967-3cdef2136999 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ca744f3dd61477b9fe67e4355e28c76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.022125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 925/1856] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:03 2022] DELETE /identity/v3/users/0ca744f3dd61477b9fe67e4355e28c76 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:04.026532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:04.026838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:04.027110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:04.037218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9e8091bc8824c29be0a285767a3d32d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.044837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9e8091bc8824c29be0a285767a3d32d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.046984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:04.048193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:04.053626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:04.058634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9e8091bc8824c29be0a285767a3d32d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.059596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9e8091bc8824c29be0a285767a3d32d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.069092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-72b40350-e81d-4a3c-a20e-2931f3785819 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9e8091bc8824c29be0a285767a3d32d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:04.076583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 932/1857] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:03 2022] DELETE /identity/v3/users/d9e8091bc8824c29be0a285767a3d32d => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:04.080903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:04.081256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:04.081529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:04.103102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:04.104367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:04.111927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 926/1858] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:04.124235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:04.128019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:04.169411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 933/1859] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:04.179823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:04.187792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8bc00c-1946-4e4a-ac8a-784595eed09f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:04.236186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad851f-36a7-4a69-9069-06553fb4c578 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:04.240646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:04.241462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:04.241695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:04.241938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:04.242310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:04.242415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:04.242611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:04.242886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:04.253056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:04.255168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:04.256913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:04.258470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:04.265887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 927/1860] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:36:04.271543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 934/1861] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:36:04.762619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:04.764137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:04.836235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a436f2d-a8dd-4897-8cb1-08e763f4cebc service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:04.839905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:04.840587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:04.841105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:04.841438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:04.843672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:04.880600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:04.952504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:04.954061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:04.970783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 928/1862] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:05.333614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:05.341405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:05.362697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:05.365533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:05.378986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216eec0e-ce31-4c08-a698-7682d890ed98 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:05.389076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:05.390631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:05.391837 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:05.392341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] PATH_INFO: `/v3/projects/97144851313240fd838b173918b5a299` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:05.395696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] RBAC: Authorizing `identity:delete_project(project_id=97144851313240fd838b173918b5a299)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:05.397784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57642e7-3837-4e6b-a3d7-83614a4fc625 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:05.404393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:05.413400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:05.416066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:05.422788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:05.424448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] PATH_INFO: `/v3/projects/ff968fce8939452880a6350ca7218e96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:05.428233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff968fce8939452880a6350ca7218e96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:05.432056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:05.467758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:05.503429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97144851313240fd838b173918b5a299', 'request_id': 'req-dd920327-da54-4272-b4e9-bd1450f69dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.509469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff968fce8939452880a6350ca7218e96', 'request_id': 'req-07396587-54c2-4142-be10-a7423a6dea3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.511921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97144851313240fd838b173918b5a299', 'request_id': 'req-dd920327-da54-4272-b4e9-bd1450f69dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.517290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff968fce8939452880a6350ca7218e96', 'request_id': 'req-07396587-54c2-4142-be10-a7423a6dea3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.521186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97144851313240fd838b173918b5a299', 'request_id': 'req-dd920327-da54-4272-b4e9-bd1450f69dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.526798 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff968fce8939452880a6350ca7218e96', 'request_id': 'req-07396587-54c2-4142-be10-a7423a6dea3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.538333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] The token cache is being invalidate because project 97144851313240fd838b173918b5a299 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:05.538961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.539932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.540763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] The token cache is being invalidate because project ff968fce8939452880a6350ca7218e96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:05.541149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.541867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 935/1863] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:05 2022] DELETE /identity/v3/projects/97144851313240fd838b173918b5a299 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:05.542005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.543733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 929/1864] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:05 2022] DELETE /identity/v3/projects/ff968fce8939452880a6350ca7218e96 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:05.545923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:05.547456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:05.548606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:05.551039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:05.633098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd920327-da54-4272-b4e9-bd1450f69dda admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:05.636681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396587-54c2-4142-be10-a7423a6dea3b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:05.639379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:05.640164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:05.640444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:05.640914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] PATH_INFO: `/v3/users/16f6a23bf79f48e4a59ae91a55501e59` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:05.642049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:05.642474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] RBAC: Authorizing `identity:delete_user(user_id=16f6a23bf79f48e4a59ae91a55501e59)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:05.642633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:05.642916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:05.643185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] PATH_INFO: `/v3/users/5ff8c5da68aa40ce8e21e12917f723f7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:05.644440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ff8c5da68aa40ce8e21e12917f723f7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:05.645756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:05.647194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:05.717881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ff8c5da68aa40ce8e21e12917f723f7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.731410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:05.744328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ff8c5da68aa40ce8e21e12917f723f7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.748856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16f6a23bf79f48e4a59ae91a55501e59'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.763776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16f6a23bf79f48e4a59ae91a55501e59'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.769486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ff8c5da68aa40ce8e21e12917f723f7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.770738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ff8c5da68aa40ce8e21e12917f723f7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.774290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ff8c5da68aa40ce8e21e12917f723f7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.817030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16f6a23bf79f48e4a59ae91a55501e59'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.820098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16f6a23bf79f48e4a59ae91a55501e59'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.822900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 930/1865] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:05 2022] DELETE /identity/v3/users/5ff8c5da68aa40ce8e21e12917f723f7 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:05.824280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:05.830365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:05.846339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16f6a23bf79f48e4a59ae91a55501e59'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:05.859656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 936/1866] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:05 2022] DELETE /identity/v3/users/16f6a23bf79f48e4a59ae91a55501e59 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:06.031677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b6207-5581-46a4-a4ee-32541454786d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:06.117482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:06.118369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.118784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.119153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.122204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.145429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:06.148020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.231226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:06.274871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:06.277744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2091814-13fd-4813-b7b3-6ee881f4f429 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:06.280310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7d0c3-31e1-4b12-b65e-65b8bdfb6339 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:06.287314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:06.288095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.288371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.288739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] PATH_INFO: `/v3/users/5c835f8463d2466cad9943bf78b8a662` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.290166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c835f8463d2466cad9943bf78b8a662)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:06.291764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 931/1867] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:05 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:06.292288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:06.371882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c835f8463d2466cad9943bf78b8a662'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.411616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c835f8463d2466cad9943bf78b8a662'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.434730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c835f8463d2466cad9943bf78b8a662'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.435971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c835f8463d2466cad9943bf78b8a662'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.451764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c835f8463d2466cad9943bf78b8a662'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.464121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 937/1868] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:06 2022] DELETE /identity/v3/users/5c835f8463d2466cad9943bf78b8a662 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:06.468539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.468877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.469369 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.521242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:06.526981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.567637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:06.570339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.664310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:06.711748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 932/1869] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:06.713108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:06.714606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.729123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e447612-38ba-4baf-a0f6-3a06140d423d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:06.735304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:06.735944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.736219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.736496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] PATH_INFO: `/v3/projects/8a67c38974c94779935c10af81cc11aa` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.737700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a67c38974c94779935c10af81cc11aa)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:06.739991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:06.752757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561046a6-1ed8-4ea0-8ac9-c5ace79ae0a8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:06.759274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:06.760640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.760926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.761201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] PATH_INFO: `/v3/projects/2728f037c73f4876a79b6480b9e1ebd9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.762648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2728f037c73f4876a79b6480b9e1ebd9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:06.766002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:06.837693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a67c38974c94779935c10af81cc11aa', 'request_id': 'req-2fdae286-867c-4f7a-87e2-fa46017104e3'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.846540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a67c38974c94779935c10af81cc11aa', 'request_id': 'req-2fdae286-867c-4f7a-87e2-fa46017104e3'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.858710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a67c38974c94779935c10af81cc11aa', 'request_id': 'req-2fdae286-867c-4f7a-87e2-fa46017104e3'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.862008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2728f037c73f4876a79b6480b9e1ebd9', 'request_id': 'req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.865159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:06.876074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] The token cache is being invalidate because project 8a67c38974c94779935c10af81cc11aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:06.876469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.877600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.883651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 933/1870] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:06 2022] DELETE /identity/v3/projects/8a67c38974c94779935c10af81cc11aa => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:06.885407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:06.887051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:06.911893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2728f037c73f4876a79b6480b9e1ebd9', 'request_id': 'req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.935038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2728f037c73f4876a79b6480b9e1ebd9', 'request_id': 'req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.952515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] The token cache is being invalidate because project 2728f037c73f4876a79b6480b9e1ebd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:06.952877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.953878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a1fe8a-5f22-45f8-98f8-e961d256a1b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:06.958696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 938/1871] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:06 2022] DELETE /identity/v3/projects/2728f037c73f4876a79b6480b9e1ebd9 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:06.960233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffa4d0-d705-4f4f-9994-1339905c2384 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:06.960521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffa4d0-d705-4f4f-9994-1339905c2384 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:06.960798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ffa4d0-d705-4f4f-9994-1339905c2384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:06.979330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdae286-867c-4f7a-87e2-fa46017104e3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:06.992059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-24ffa4d0-d705-4f4f-9994-1339905c2384 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:06.993937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-24ffa4d0-d705-4f4f-9994-1339905c2384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.050479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.051820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.052206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.052469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.054330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.090044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 939/1872] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:07.091551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7302c078-8f49-496e-b64e-804e6d775539 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.091696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7302c078-8f49-496e-b64e-804e6d775539 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.091886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7302c078-8f49-496e-b64e-804e6d775539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.100512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:07.127115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7302c078-8f49-496e-b64e-804e6d775539 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:07.129025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7302c078-8f49-496e-b64e-804e6d775539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.141612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 940/1873] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:07.143512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.144015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.144524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.171215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.174846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a8ae69-1138-4247-ad03-b32fdb5a62cb service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.179073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:07.181918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.188874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 934/1874] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:07.190360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.190933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.191471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.195486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 941/1875] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:07.199840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.213946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5d7d7a-66c2-40a4-87a8-b7a1ee69d03a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.218778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:07.220434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.221749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.224276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.225269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.226114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.227006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:07.227452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.231653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.236244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 935/1876] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:07.244145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.252318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 942/1877] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:07.257003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.261491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77ad543-5b13-4eea-b5f3-3129f2570b7a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.267402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.268077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.268357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.268630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.269286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:07.273029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b1ddf7-bc50-49a6-b3d1-dae0e2ea3ae9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.286826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.286826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.288451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.289092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.289396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.289639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.291235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.292510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.313352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 936/1878] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:07.314829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.330088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e782f4-4f23-4dc3-9d30-485ad9db4f8b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.336780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.337485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.337865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.338204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.339035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.341343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.347677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 943/1879] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/projects => generated 357 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:07.352535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.405285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 937/1880] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/projects => generated 361 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:07.406354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.501621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602d6b61-9d74-47c5-993b-5ded569b121d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.547696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede3f220-7ac1-4d88-a3bd-11187644e2f3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.558406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.563629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.571477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.572038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.573458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.575874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.604309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:07.630107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 938/1881] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/users => generated 281 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:07.634879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.635513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.635788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.636051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.636366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.636694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.639829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.657533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3aa15-1d81-438d-b373-7d763a561ed0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.673096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.674097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.674534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.676890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.677385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 944/1882] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:07 2022] POST /identity/v3/users => generated 279 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:07.678244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.681479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.684023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.701458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6170d2cd-2f54-4d00-b20f-7d90ba592baf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.707094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 939/1883] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:07.711422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.713642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.715133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.715756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.716383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.717544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.719592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.733637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a96769-c008-4602-a1a7-7b044e7495d8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.738505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 945/1884] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:07.740741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.742728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:07.756701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.757331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.757603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.757862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.758509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.760475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.769400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e9aebf-e7de-4dc4-9a4f-0e087d2760d4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.777677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.790603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 940/1885] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:07.790830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.795281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.795833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.796400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] PATH_INFO: `/v3/projects/a00e35a1b97b49f6b503517fcfba3a3d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.798257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a00e35a1b97b49f6b503517fcfba3a3d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.800402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.809983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e21788-b172-421b-bc7c-da6a0313a3a6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.823514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.824297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.825163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.826458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.827926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.829904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.849869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 941/1886] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:07.851036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.866297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a130fcd8-5ba5-4124-bab5-b893843283c0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:07.873434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:07.874729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:07.875279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:07.875490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] PATH_INFO: `/v3/projects/65460c9571604e9fa131786172b0fa70/users/e093951c7aba4524b3bcd9c263bf7726/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:07.906158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65460c9571604e9fa131786172b0fa70, user_id=e093951c7aba4524b3bcd9c263bf7726, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:07.912506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:07.917613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a00e35a1b97b49f6b503517fcfba3a3d', 'request_id': 'req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:07.934851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 942/1887] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:07 2022] PUT /identity/v3/projects/65460c9571604e9fa131786172b0fa70/users/e093951c7aba4524b3bcd9c263bf7726/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:07.942788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:07.942937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a00e35a1b97b49f6b503517fcfba3a3d', 'request_id': 'req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:07.967565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:07.968644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a00e35a1b97b49f6b503517fcfba3a3d', 'request_id': 'req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.009183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] The token cache is being invalidate because project a00e35a1b97b49f6b503517fcfba3a3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:08.014821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.033687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c41ff274-d9ee-4bc5-8aa8-27752753fc5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.042930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 946/1888] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:07 2022] DELETE /identity/v3/projects/a00e35a1b97b49f6b503517fcfba3a3d => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:08.042930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb638f-81d6-4e39-adce-45f0725f7994 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.042930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb638f-81d6-4e39-adce-45f0725f7994 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.042930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb638f-81d6-4e39-adce-45f0725f7994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.137668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1fcb638f-81d6-4e39-adce-45f0725f7994 None None] MFA Rules not processed for user `e093951c7aba4524b3bcd9c263bf7726`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:08.145257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec8f4b3-a6e9-41f7-9695-f41edf1c8337 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.159486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1fcb638f-81d6-4e39-adce-45f0725f7994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.162003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.165429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.165776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.168107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] PATH_INFO: `/v3/projects/8ce4cbdc153840b0a42c8516809391d5/users/6d0073b1a9074106bd70eaa5f6d41835/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.210644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ce4cbdc153840b0a42c8516809391d5, user_id=6d0073b1a9074106bd70eaa5f6d41835, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.213541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1e920-001b-4346-bab3-03bdd12ac9a5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.252091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 943/1889] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:07 2022] PUT /identity/v3/projects/8ce4cbdc153840b0a42c8516809391d5/users/6d0073b1a9074106bd70eaa5f6d41835/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:08.267266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.270370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.270745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.296632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 947/1890] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:08.306384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564cfbb-010b-41f0-875c-9cf3c32cb5b9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.306697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564cfbb-010b-41f0-875c-9cf3c32cb5b9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.306983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7564cfbb-010b-41f0-875c-9cf3c32cb5b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.322846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:08.324242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.362082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7564cfbb-010b-41f0-875c-9cf3c32cb5b9 None None] MFA Rules not processed for user `6d0073b1a9074106bd70eaa5f6d41835`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:08.364213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7564cfbb-010b-41f0-875c-9cf3c32cb5b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.384948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:08.421627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 944/1891] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:08.423510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.431440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.471681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 948/1892] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:08.471681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.471681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.471681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.496409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:08.497740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.513784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 949/1893] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:08.513784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.514442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.532523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f769289-b92e-41f4-9cb4-5d0421e359b6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.539095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.539850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.540459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.540731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.541370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.542916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043728b-b447-43be-988b-c06b7b3e5e5d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.543521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.547858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.549282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.549691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.550174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.584165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 950/1894] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/projects => generated 357 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:08.585343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.650912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:08.674924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.682024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.690000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9795e5f-d03a-48d8-9148-287d12ce8181 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.699291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.699919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.700196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.700469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.700570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 945/1895] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:08 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:08.703187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.710744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:08.711058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.713288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.731298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae61f216-4093-4b33-8cb7-dbfefa9cbf2c service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.735168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 951/1896] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:08.736584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.738130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.739024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.740202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.740500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.741552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.742180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.744677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.756134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82825a94-c4e6-4d2f-875f-57641a5e239d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.761586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.762536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.762865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.763234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] PATH_INFO: `/v3/users/e093951c7aba4524b3bcd9c263bf7726` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.764664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e093951c7aba4524b3bcd9c263bf7726)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.783684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.796313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 946/1897] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/users => generated 285 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:08.797607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.818321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a6f81-54f5-495c-bf32-732b7e3e2825 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:08.827035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:08.828084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:08.828376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:08.828696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:08.829706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:08.833255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:08.848594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e093951c7aba4524b3bcd9c263bf7726'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.858661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e093951c7aba4524b3bcd9c263bf7726'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.875268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e093951c7aba4524b3bcd9c263bf7726'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.876145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e093951c7aba4524b3bcd9c263bf7726'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.882402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 947/1898] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/projects => generated 375 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:08.882402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.883985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.887097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e093951c7aba4524b3bcd9c263bf7726'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:08.888989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:08.899774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 952/1899] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:08 2022] DELETE /identity/v3/users/e093951c7aba4524b3bcd9c263bf7726 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:08.900766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:08.903633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:08.976558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4107b-de0e-442e-a3cd-355563323185 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.033071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ac5fb7-bf2b-4ec2-b762-eb68823622f7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.061723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.062503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.064407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.064795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.066189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.071793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.072415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.073895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.074177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.074448 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.075862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.078089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e321c58d-e5c9-4cb0-acaa-6440e56b3825 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.091890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 948/1900] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:08 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:09.093928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.094455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.095079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.122674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 953/1901] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:08 2022] POST /identity/v3/users => generated 288 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:09.124003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.125401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:09.127166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:09.142601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5340c-d710-42f5-b37e-4ba5d4274a3d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.148995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.148995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.148995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.148995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] PATH_INFO: `/v3/projects/9a7221a9e35d43a88520b8ad84061096/users/542bb58e322e41e5a0e8586732e50f3e/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.162984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 949/1902] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:09.167980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.172128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a7221a9e35d43a88520b8ad84061096, user_id=542bb58e322e41e5a0e8586732e50f3e, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.175137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.191466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebaa43b-9d95-41cc-9489-ecb5841c8357 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.195526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 954/1903] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:09 2022] PUT /identity/v3/projects/9a7221a9e35d43a88520b8ad84061096/users/542bb58e322e41e5a0e8586732e50f3e/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:09.198700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.199821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:09.275889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d69022-f67d-4173-8fb1-cca665265387 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.280303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.281715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.282104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.282383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.283074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:09.307012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.307963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.308379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.309278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.310256 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.313114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.329013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 950/1904] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:09 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:09.330499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.343883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280e8ec-436c-45ed-898b-9de188471927 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.350010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.350510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.354952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.355406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.357316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.360028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.374892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 951/1905] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:09 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:09.375656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.405979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd0123-ec01-4703-a647-4e3e79aa7517 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.406097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.406097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.407673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.409012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.409012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.409012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.409733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.412877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.416512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 955/1906] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:09.417331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.428676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ca429d-e0b1-4725-96a1-101be2c56c3d service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.432644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 952/1907] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:09 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:09.433718 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.434402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.434714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.434987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] PATH_INFO: `/v3/projects/9a7221a9e35d43a88520b8ad84061096/users/542bb58e322e41e5a0e8586732e50f3e/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.436944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a7221a9e35d43a88520b8ad84061096, user_id=542bb58e322e41e5a0e8586732e50f3e, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.439706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5c8fba-17fd-4d90-99e6-aa85152576c1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.447526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:09.451235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.458598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 956/1908] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:09 2022] PUT /identity/v3/projects/9a7221a9e35d43a88520b8ad84061096/users/542bb58e322e41e5a0e8586732e50f3e/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:09.466802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33139-082e-402b-ac55-47a09759c444 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.467209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33139-082e-402b-ac55-47a09759c444 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.467585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a33139-082e-402b-ac55-47a09759c444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.478414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7688eeec-d40b-4dd1-8df9-bd82e96d7d48 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.542994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-76a33139-082e-402b-ac55-47a09759c444 None None] MFA Rules not processed for user `542bb58e322e41e5a0e8586732e50f3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:09.544648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-76a33139-082e-402b-ac55-47a09759c444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:09.620619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:09.621585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.621864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.622271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] PATH_INFO: `/v3/projects/4cffcc28402548458774bff31a89deec/users/d74e552d274e405da239f233ed3bc77d/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.664577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cffcc28402548458774bff31a89deec, user_id=d74e552d274e405da239f233ed3bc77d, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:09.666604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 957/1909] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:09 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:09.674811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2613e-9f7f-429b-b02f-bae71ef49195 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:09.702667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a33139-082e-402b-ac55-47a09759c444 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:09.709634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-76a33139-082e-402b-ac55-47a09759c444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:09.710617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 953/1910] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:09 2022] PUT /identity/v3/projects/4cffcc28402548458774bff31a89deec/users/d74e552d274e405da239f233ed3bc77d/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:09.721111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:09.721426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:09.721663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:09.814358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] MFA Rules not processed for user `d74e552d274e405da239f233ed3bc77d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:09.818811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:09.869793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a33139-082e-402b-ac55-47a09759c444 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:09.954156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 954/1911] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:10.008683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:10.009411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.009790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.010131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.104752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:10.107873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.127584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:10.148604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:10.153746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:10.192808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 958/1912] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:09 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:10.222051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:10.230426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:10.231763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.248062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d885d2-af8c-43e0-81d0-a466ea8f493e service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:10.261880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:10.262524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.262830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.263098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.288657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:10.296746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:10.301040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707bf15-f7f8-4f43-9fb2-f87bb82b8b90 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:10.312515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 959/1913] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:10.326211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:10.326932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.327207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.327477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] PATH_INFO: `/v3/projects/65460c9571604e9fa131786172b0fa70` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.331646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65460c9571604e9fa131786172b0fa70)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:10.347164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:10.453610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65460c9571604e9fa131786172b0fa70', 'request_id': 'req-81e203b6-2cdf-475b-bfae-180af8b61477'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.487226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65460c9571604e9fa131786172b0fa70', 'request_id': 'req-81e203b6-2cdf-475b-bfae-180af8b61477'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.515141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65460c9571604e9fa131786172b0fa70', 'request_id': 'req-81e203b6-2cdf-475b-bfae-180af8b61477'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.527516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:10.529563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.548099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c431b-64bd-414b-82c2-55f2734425d2 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:10.563559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] The token cache is being invalidate because project 65460c9571604e9fa131786172b0fa70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:10.565806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.567396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.571769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 955/1914] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:10 2022] DELETE /identity/v3/projects/65460c9571604e9fa131786172b0fa70 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:10.575456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:10.578135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.685047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:10.687397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.688009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.688634 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] PATH_INFO: `/v3/users/e6fe45d17ef64333822b847bc51c1d24` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.691193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6fe45d17ef64333822b847bc51c1d24)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:10.694532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:10.725795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e203b6-2cdf-475b-bfae-180af8b61477 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:10.729898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:10.730858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.731359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.731743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.733936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.779704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6fe45d17ef64333822b847bc51c1d24'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.797204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6fe45d17ef64333822b847bc51c1d24'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.812578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:10.814386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185ae4b5-39aa-4a9d-8b2a-120f49086033 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:10.824341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6fe45d17ef64333822b847bc51c1d24'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.825966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6fe45d17ef64333822b847bc51c1d24'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.828458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 956/1915] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:10 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:10.829905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea937f7-a08b-40b4-9338-02e591894564 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.830425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea937f7-a08b-40b4-9338-02e591894564 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.830942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea937f7-a08b-40b4-9338-02e591894564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.860983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3ea937f7-a08b-40b4-9338-02e591894564 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:10.862534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3ea937f7-a08b-40b4-9338-02e591894564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.865335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6e86176a-1f97-4c34-9d5b-b9ee6e2ee328 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6fe45d17ef64333822b847bc51c1d24'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:10.881496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 957/1916] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:10.884992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.885523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.885903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.891174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 960/1917] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:10 2022] DELETE /identity/v3/users/e6fe45d17ef64333822b847bc51c1d24 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:10.892608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:10.895552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:10.895934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:10.932422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:10.939008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:10.941537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.943111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:10.964078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:11.067530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 958/1918] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:11.071548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.082474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c350b27a-1a67-4640-b2e7-e9f874983906 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.085430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:11.091317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.092346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.092706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.093120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.094287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:11.094610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.096945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.111335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 961/1919] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:11.113054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 959/1920] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:11.118967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.120517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.122311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:11.131329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1d692-0d86-4b79-9854-f8c98a6e6e57 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.139834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.140461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.140743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.141015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.141668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.144137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.195155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 962/1921] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:11 2022] POST /identity/v3/projects => generated 363 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:11.199833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.226260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1f2361-4a18-46f7-9bb6-c26046a6d996 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.289404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.290306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.290813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.291351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.328576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39336009-2cca-431d-92b1-6551e05d2497 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.335645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.337176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.337604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.338016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.338872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.342139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.380966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 963/1922] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:11 2022] POST /identity/v3/users => generated 282 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:11.382117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.391023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.393003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.394487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf13b14-9af1-46f5-bc6e-a82f475a0571 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.402207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.403574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.403922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.404508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.405968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.408732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.409188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 960/1923] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:11.423934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 964/1924] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:11 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:11.426681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.442110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737a4b5b-e91b-477f-a75c-a1f798cd586c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.448331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.449200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.449693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.450135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.450999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.453194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd39441-d753-4d88-b87a-e76e65746914 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.455356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:11.471219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 961/1925] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:11 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:11.477730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.492855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92ae28-a5f1-4a1c-9028-88d92ac07793 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.499127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.500091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.500683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.501120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] PATH_INFO: `/v3/projects/3bd60df9238f4c1689138e1c28e74eca/users/75b28992f2034e40b5df5db29d321a46/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.534917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bd60df9238f4c1689138e1c28e74eca, user_id=75b28992f2034e40b5df5db29d321a46, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.539187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.582826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 965/1926] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:11 2022] PUT /identity/v3/projects/3bd60df9238f4c1689138e1c28e74eca/users/75b28992f2034e40b5df5db29d321a46/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:11.589075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.589075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.589365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.621910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] MFA Rules not processed for user `75b28992f2034e40b5df5db29d321a46`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:11.624060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:11.770158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 962/1927] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:11 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:11.779546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.780991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:11.898203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a4a80-8079-45e7-a245-65c6ba80432f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:11.905726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:11.906124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:11.906960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:11.907203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:11.907544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:11.909013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:11.918422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:11.920147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:11.930803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 966/1928] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:11 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:12.020651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfeea7c7-f6e3-4d90-997f-c7da09a600ff None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:12.026427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:12.027656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:12.028078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:12.028640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] PATH_INFO: `/v3/projects/d9e60959d3794ad1b9a88e15893a453a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:12.030323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9e60959d3794ad1b9a88e15893a453a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:12.033289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:12.116464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9e60959d3794ad1b9a88e15893a453a', 'request_id': 'req-856de980-05df-445a-acc5-2eb5226810f5'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.158350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9e60959d3794ad1b9a88e15893a453a', 'request_id': 'req-856de980-05df-445a-acc5-2eb5226810f5'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.178451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9e60959d3794ad1b9a88e15893a453a', 'request_id': 'req-856de980-05df-445a-acc5-2eb5226810f5'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.205900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] The token cache is being invalidate because project d9e60959d3794ad1b9a88e15893a453a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:12.206455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.209643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.211611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 963/1929] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:11 2022] DELETE /identity/v3/projects/d9e60959d3794ad1b9a88e15893a453a => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:12.215685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:12.232970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76df572a-4ddb-41ce-8ec7-00915e4ecd51 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:12.394064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:12.484933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:12.485688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:12.485968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:12.486247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] PATH_INFO: `/v3/users/4166dfecb3a547e79050d4ef29720551` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:12.488201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4166dfecb3a547e79050d4ef29720551)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:12.497518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:12.659811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4166dfecb3a547e79050d4ef29720551'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.670199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4166dfecb3a547e79050d4ef29720551'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.704286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4166dfecb3a547e79050d4ef29720551'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.705111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4166dfecb3a547e79050d4ef29720551'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.717111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4166dfecb3a547e79050d4ef29720551'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:12.734366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 967/1930] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:12 2022] DELETE /identity/v3/users/4166dfecb3a547e79050d4ef29720551 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:13.233338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:13.234883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:13.248136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856de980-05df-445a-acc5-2eb5226810f5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:13.261622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:13.312797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:13.313462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:13.313814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:13.314198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] PATH_INFO: `/v3/projects/cbda626d8c314a21b56d943b78d13f94` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:13.315511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbda626d8c314a21b56d943b78d13f94)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:13.317806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:13.370638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbda626d8c314a21b56d943b78d13f94', 'request_id': 'req-303d30be-a412-4e10-8cc3-b8bb49f16a8d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.377569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbda626d8c314a21b56d943b78d13f94', 'request_id': 'req-303d30be-a412-4e10-8cc3-b8bb49f16a8d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.388031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbda626d8c314a21b56d943b78d13f94', 'request_id': 'req-303d30be-a412-4e10-8cc3-b8bb49f16a8d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.400354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] The token cache is being invalidate because project cbda626d8c314a21b56d943b78d13f94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:13.400648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.401335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.402923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 964/1931] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:13 2022] DELETE /identity/v3/projects/cbda626d8c314a21b56d943b78d13f94 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:13.407742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:13.409105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:13.422890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e419f8-9101-4929-baf3-1f95c5cd4b55 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:13.484827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:13.485652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:13.486115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:13.486403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] PATH_INFO: `/v3/users/ae9ef579ec204a6abdf7f28ef00e3f9c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:13.487861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae9ef579ec204a6abdf7f28ef00e3f9c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:13.490250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:13.542719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae9ef579ec204a6abdf7f28ef00e3f9c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.547852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:13.549895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae9ef579ec204a6abdf7f28ef00e3f9c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.563836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae9ef579ec204a6abdf7f28ef00e3f9c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.564560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae9ef579ec204a6abdf7f28ef00e3f9c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.572786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0f62ef55-65f2-40fa-bed4-98d227ac10d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae9ef579ec204a6abdf7f28ef00e3f9c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:13.579104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 968/1932] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:13 2022] DELETE /identity/v3/users/ae9ef579ec204a6abdf7f28ef00e3f9c => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:13.907568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:13.909032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:13.924601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d30be-a412-4e10-8cc3-b8bb49f16a8d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:13.992442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:13.993098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:13.993376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:13.993646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] PATH_INFO: `/v3/projects/541fdcf6bac045f2ac69d9e46307d590` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:13.994982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] RBAC: Authorizing `identity:delete_project(project_id=541fdcf6bac045f2ac69d9e46307d590)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:13.996889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.046170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:14.052625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '541fdcf6bac045f2ac69d9e46307d590', 'request_id': 'req-b7a54477-03d7-4db9-9c91-13dce7850996'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:14.061195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '541fdcf6bac045f2ac69d9e46307d590', 'request_id': 'req-b7a54477-03d7-4db9-9c91-13dce7850996'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:14.071646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '541fdcf6bac045f2ac69d9e46307d590', 'request_id': 'req-b7a54477-03d7-4db9-9c91-13dce7850996'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:14.086327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] The token cache is being invalidate because project 541fdcf6bac045f2ac69d9e46307d590 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:14.086649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:14.087341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b7a54477-03d7-4db9-9c91-13dce7850996 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:14.088857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 965/1933] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:13 2022] DELETE /identity/v3/projects/541fdcf6bac045f2ac69d9e46307d590 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:14.093682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.094506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.095746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.118692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:14.119921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:14.216447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d797c5-a348-4357-8613-983b49535143 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.216723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d797c5-a348-4357-8613-983b49535143 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.216983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d797c5-a348-4357-8613-983b49535143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.217748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 969/1934] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:14.238229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-14d797c5-a348-4357-8613-983b49535143 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:14.239984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-14d797c5-a348-4357-8613-983b49535143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:14.247885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 966/1935] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:14.252580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.263537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e3c2b4-5a95-45b6-843e-c837c0a2c0a6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.268172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.268789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.269063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.269338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.270752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:14.271284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.272908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.289777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 970/1936] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:36:14.291202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d797c5-a348-4357-8613-983b49535143 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.300677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d797c5-a348-4357-8613-983b49535143 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.306185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.306820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.307095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.307364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.308030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.309925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.343195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 967/1937] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:14 2022] POST /identity/v3/projects => generated 391 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:14.344621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.355354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f9429-fdd4-4b3f-a06a-b10fa6119bc0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.423539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.424169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.424450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.424726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.425380 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.427623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.454092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 971/1938] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:14 2022] POST /identity/v3/users => generated 296 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:14.457960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.472143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e1f72-3e45-4a56-b4dd-3ec86c836e98 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.479389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.480362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.480820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.481272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.482358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.485250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.500623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 968/1939] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:14.502036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.516589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea287e8-315a-43d6-9e2d-b6d48e8aa7b0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.526175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.526823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.527104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.527378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.528060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.529671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4da5e-f084-47ef-8905-049887693f6c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.542199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 972/1940] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:14.546381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.556211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14d8fc-08b9-4d3a-9ca8-2a6d6775bbdb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.560917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.561576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.561853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.562126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] PATH_INFO: `/v3/projects/00fc832a0f174eb098540d14cb9d90ec/users/bde6af2d72764d2784a82a72663c4dc1/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.577697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00fc832a0f174eb098540d14cb9d90ec, user_id=bde6af2d72764d2784a82a72663c4dc1, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.580406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.595107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 969/1941] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:14 2022] PUT /identity/v3/projects/00fc832a0f174eb098540d14cb9d90ec/users/bde6af2d72764d2784a82a72663c4dc1/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:14.605441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.605817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.606301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.636574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] MFA Rules not processed for user `bde6af2d72764d2784a82a72663c4dc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:14.637934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:14.665497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:14.707948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 973/1942] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:14.719146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:14.720864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:14.792545 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ff7-5d66-4e9b-ac01-5159fbd03b92 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:14.797086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:14.797724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:14.797999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:14.798270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:14.810850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:14.812289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd74e8ba-720f-4448-ae79-d1058217c534 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:14.824568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 970/1943] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:14 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:24.704310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=15 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:24.819247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=60,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:41.607734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:41.719852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9885316-da64-478b-b6b3-3defdeb02fcb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:41.726181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:41.727038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:41.727395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:41.727743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] PATH_INFO: `/v3/users/bde6af2d72764d2784a82a72663c4dc1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:41.729881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bde6af2d72764d2784a82a72663c4dc1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:41.732243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:41.827109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bde6af2d72764d2784a82a72663c4dc1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:41.845332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bde6af2d72764d2784a82a72663c4dc1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:41.866674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bde6af2d72764d2784a82a72663c4dc1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:41.867720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bde6af2d72764d2784a82a72663c4dc1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:41.877891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bde6af2d72764d2784a82a72663c4dc1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:41.886992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 974/1944] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:41 2022] DELETE /identity/v3/users/bde6af2d72764d2784a82a72663c4dc1 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:41.894785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:41.894785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:41.894785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:41.921054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:41.923195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.036519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 971/1945] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:42.047703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:42.052454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.156597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:42.159169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.170753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0d470-300e-4b6f-9105-c18b2d749195 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:42.174849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:42.176370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.176852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.177192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.177863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d3b08a-2a53-4f51-8bab-6f84b43d56fe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:42.184745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:42.185577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.186442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.186931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] PATH_INFO: `/v3/users/6d0073b1a9074106bd70eaa5f6d41835` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.188676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d0073b1a9074106bd70eaa5f6d41835)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:42.189213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:42.191934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:42.192139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1b84b4-d530-41c7-8217-76124c17b0ae service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:42.203341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 975/1946] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:42.276939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d0073b1a9074106bd70eaa5f6d41835'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.288580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d0073b1a9074106bd70eaa5f6d41835'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.310364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d0073b1a9074106bd70eaa5f6d41835'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.311366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d0073b1a9074106bd70eaa5f6d41835'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.312042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d0073b1a9074106bd70eaa5f6d41835'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.342911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 972/1947] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:42 2022] DELETE /identity/v3/users/6d0073b1a9074106bd70eaa5f6d41835 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:42.345871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.346397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.347077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.395499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:42.406282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.453683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:42.566444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 976/1948] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:42.575920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:42.577114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.660955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:42.662700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.672705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286a9d41-15e1-45c2-9ce9-39c91342eedc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:42.679571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:42.680912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.681385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.681836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.689950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f7ed53-d8aa-4d8d-9eb1-4f89ad9d4172 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:42.697057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:42.697758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.698044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.698352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] PATH_INFO: `/v3/projects/00fc832a0f174eb098540d14cb9d90ec` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.699528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00fc832a0f174eb098540d14cb9d90ec)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:42.702362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:42.702692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:42.705671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08c4656-27fe-4dbd-8192-83f0f039e89a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:42.719205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 973/1949] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:42.783567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00fc832a0f174eb098540d14cb9d90ec', 'request_id': 'req-d41a1df3-855e-4f82-a3cc-27ff23d34e09'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.791432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00fc832a0f174eb098540d14cb9d90ec', 'request_id': 'req-d41a1df3-855e-4f82-a3cc-27ff23d34e09'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.805160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00fc832a0f174eb098540d14cb9d90ec', 'request_id': 'req-d41a1df3-855e-4f82-a3cc-27ff23d34e09'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.830883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] The token cache is being invalidate because project 00fc832a0f174eb098540d14cb9d90ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:42.832448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.833953 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d41a1df3-855e-4f82-a3cc-27ff23d34e09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:42.836084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 977/1950] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:42 2022] DELETE /identity/v3/projects/00fc832a0f174eb098540d14cb9d90ec => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:42.845996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417991f-6870-4d6d-9755-77034b186331 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:42.846261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417991f-6870-4d6d-9755-77034b186331 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:42.846721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417991f-6870-4d6d-9755-77034b186331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:42.882883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3417991f-6870-4d6d-9755-77034b186331 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:42.899192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3417991f-6870-4d6d-9755-77034b186331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:42.905140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:43.047820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 974/1951] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:43.053317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.053720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.053888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.092426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:43.094861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.114611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 978/1952] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:43.118738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3417991f-6870-4d6d-9755-77034b186331 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.137839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3417991f-6870-4d6d-9755-77034b186331 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.149565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.149845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.150119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.150388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.151205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:43.151618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.153666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.173297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 975/1953] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:36:43.180834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.202407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a5ebdd-368b-445a-9cb5-eeccc0081cbc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.207197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.215193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.216359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.216451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.216592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.216805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.217514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.228262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.241319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df52f19-9d58-4f22-b098-7681af6df89e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.249825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.250905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.251586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.251952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] PATH_INFO: `/v3/projects/8ce4cbdc153840b0a42c8516809391d5` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.254227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ce4cbdc153840b0a42c8516809391d5)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.258346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.266354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 979/1954] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:43 2022] POST /identity/v3/projects => generated 377 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:43.272952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.348701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ce4cbdc153840b0a42c8516809391d5', 'request_id': 'req-3ef64aea-39eb-4673-96c7-7068eb93bda2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.360426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ce4cbdc153840b0a42c8516809391d5', 'request_id': 'req-3ef64aea-39eb-4673-96c7-7068eb93bda2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.374377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ce4cbdc153840b0a42c8516809391d5', 'request_id': 'req-3ef64aea-39eb-4673-96c7-7068eb93bda2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.391940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] The token cache is being invalidate because project 8ce4cbdc153840b0a42c8516809391d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:43.392307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.396691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.401637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 976/1955] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:43 2022] DELETE /identity/v3/projects/8ce4cbdc153840b0a42c8516809391d5 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:43.403770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.405335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.411550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54003dce-d53b-41e9-9997-bc519b2b22a7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.439219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:43.488955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.489678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.490083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.490438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.491194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.493243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.508368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef64aea-39eb-4673-96c7-7068eb93bda2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.514329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.515228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.515538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.516012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] PATH_INFO: `/v3/users/542bb58e322e41e5a0e8586732e50f3e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.517395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] RBAC: Authorizing `identity:delete_user(user_id=542bb58e322e41e5a0e8586732e50f3e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.521173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.531133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 980/1956] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:43 2022] POST /identity/v3/users => generated 289 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:43.543799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.547672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.564890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044656df-f977-4bd2-b193-cd7184dd207c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.570983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.572066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.572689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.573908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.573908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.576481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.582206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:43.590889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 981/1957] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:43 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:43.595057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.607153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea3f598-4615-4f8e-a3da-a9fadf430abb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.615105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.615743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.616054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.616327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.616996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.618706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.620067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '542bb58e322e41e5a0e8586732e50f3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.635065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 982/1958] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:43.636723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.639978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '542bb58e322e41e5a0e8586732e50f3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.649894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3168404-9cdb-4e43-86e4-c828bf68c6c5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.656162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.656924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.657262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.657536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] PATH_INFO: `/v3/projects/2f94e3ae140b43739febe1111e18604b/users/ab4b3c2b3f4b4978b0eb472f9c2736b1/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.660236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '542bb58e322e41e5a0e8586732e50f3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.661272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '542bb58e322e41e5a0e8586732e50f3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.662263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-772775be-4283-4937-b3b9-fde2c359703e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '542bb58e322e41e5a0e8586732e50f3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:43.677138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f94e3ae140b43739febe1111e18604b, user_id=ab4b3c2b3f4b4978b0eb472f9c2736b1, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.680995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:43.682451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 977/1959] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:43 2022] DELETE /identity/v3/users/542bb58e322e41e5a0e8586732e50f3e => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:43.706140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 983/1960] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:43 2022] PUT /identity/v3/projects/2f94e3ae140b43739febe1111e18604b/users/ab4b3c2b3f4b4978b0eb472f9c2736b1/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:43.714632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.715118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.715668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.746210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:43.747718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.836895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 978/1961] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:43.851575 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:43.855907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:43.961460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a382d71-bbf9-42f8-88be-f23811fe2b5c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:43.969697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:43.970539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:43.971133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:43.972516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:43.984949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:43.986823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.002257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 984/1962] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:44.363443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.364937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.383194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.383631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a758a7-5465-46ca-8ef3-c40eedf252d8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.386004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.389078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.389974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.390196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.390470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] PATH_INFO: `/v3/projects/9a7221a9e35d43a88520b8ad84061096` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.391701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a7221a9e35d43a88520b8ad84061096)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.395943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.428787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab196a62-c4d5-4fc1-bada-44e325de29c8 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.440090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.442441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.443450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.444155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] PATH_INFO: `/v3/users/d74e552d274e405da239f233ed3bc77d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.448149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d74e552d274e405da239f233ed3bc77d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.452519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.512910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a7221a9e35d43a88520b8ad84061096', 'request_id': 'req-d4460933-2581-4aa1-ab2c-31a752f1bb9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.522836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a7221a9e35d43a88520b8ad84061096', 'request_id': 'req-d4460933-2581-4aa1-ab2c-31a752f1bb9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.534205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a7221a9e35d43a88520b8ad84061096', 'request_id': 'req-d4460933-2581-4aa1-ab2c-31a752f1bb9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.548979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] The token cache is being invalidate because project 9a7221a9e35d43a88520b8ad84061096 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:44.549547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.550161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4460933-2581-4aa1-ab2c-31a752f1bb9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.552478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 979/1963] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:44 2022] DELETE /identity/v3/projects/9a7221a9e35d43a88520b8ad84061096 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:44.560481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828fb1-34d6-4de0-ad03-788a1125c6a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.560819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828fb1-34d6-4de0-ad03-788a1125c6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.561095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc828fb1-34d6-4de0-ad03-788a1125c6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.579232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd74e552d274e405da239f233ed3bc77d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.599037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd74e552d274e405da239f233ed3bc77d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.600602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dc828fb1-34d6-4de0-ad03-788a1125c6a0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:44.602368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dc828fb1-34d6-4de0-ad03-788a1125c6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.621131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:44.634443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd74e552d274e405da239f233ed3bc77d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.636950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd74e552d274e405da239f233ed3bc77d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.648784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:44.658820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c7f067cd-60d0-44c5-bafe-b754ae176266 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd74e552d274e405da239f233ed3bc77d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:44.667353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 985/1964] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:44 2022] DELETE /identity/v3/users/d74e552d274e405da239f233ed3bc77d => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:44.673051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef549-d467-4536-8e20-e40627a81651 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.676135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef549-d467-4536-8e20-e40627a81651 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.676135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abef549-d467-4536-8e20-e40627a81651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.705441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9abef549-d467-4536-8e20-e40627a81651 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:44.708390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9abef549-d467-4536-8e20-e40627a81651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.731481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 980/1965] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:44.745483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.746142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.746730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.789609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:44.791048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.802565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 986/1966] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:44.817692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abef549-d467-4536-8e20-e40627a81651 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.817692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9abef549-d467-4536-8e20-e40627a81651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:44.891898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 981/1967] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:44.899594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.918732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a88e3-be66-4c90-9e36-1f950645432a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.927347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.928992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.928992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.928992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.929492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:44.929773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:44.933954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:44.961702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 982/1968] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:44.964869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:44.986640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abef549-d467-4536-8e20-e40627a81651 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.988443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4e2c35-ad4e-436f-9a07-fae9ceecdbb2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:44.996293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:44.997167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:44.997458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:44.998935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:44.998935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.000937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.001679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.003012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.003012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.003474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.061917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.062964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 983/1969] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:44 2022] POST /identity/v3/projects => generated 369 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:45.071114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.071314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.097885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 987/1970] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:45.205298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7365f917-cdd2-44b5-8179-9687e5cf9d7a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.212002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.212002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.212550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.212951 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.213881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.216299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.252861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 984/1971] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/users => generated 285 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:45.255410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.266900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872cf72a-f243-4ef3-8e33-f80752a8f25e service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.274351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.275226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.275640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.276855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.276855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.279635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.295428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 988/1972] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:45.297192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.311086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb895da6-a7ee-45b6-9581-d63dbedfe2a9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.316860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.317648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.318403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.318703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.319694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.321474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c052125-3cb7-4b1f-a058-2002199b042f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.349807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 985/1973] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:45.355174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.367405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e474367-0764-4eaa-8a90-149a39ed7f34 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.374166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.375173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.375617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.376224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] PATH_INFO: `/v3/projects/3c243001532b4a398a8ce8578522baf4/users/b6698bed548945b081d1c101b7803263/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.400164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c243001532b4a398a8ce8578522baf4, user_id=b6698bed548945b081d1c101b7803263, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.403898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.424250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 989/1974] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:45 2022] PUT /identity/v3/projects/3c243001532b4a398a8ce8578522baf4/users/b6698bed548945b081d1c101b7803263/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:45.437042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.438000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.438819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.498393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:45.512368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.515597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:45.525208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] MFA Rules not processed for user `b6698bed548945b081d1c101b7803263`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:45.535287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:45.631713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:45.664418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c62abb4-4a76-49bb-a884-ee127c0897ff admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.673033 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.674319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.675960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.676420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] PATH_INFO: `/v3/projects/4cffcc28402548458774bff31a89deec` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.679210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cffcc28402548458774bff31a89deec)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.682886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.696696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 986/1975] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:45.698146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.712801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d6c9d0-ceb3-4369-9009-61ce5cbafacd None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.719972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.722339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.723181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.723737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.725634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.729433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.790052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 987/1976] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/projects => generated 369 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:45.792682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:45.798074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cffcc28402548458774bff31a89deec', 'request_id': 'req-eed23027-10ac-41f7-a4a8-c2bacbb1483d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:45.810998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cffcc28402548458774bff31a89deec', 'request_id': 'req-eed23027-10ac-41f7-a4a8-c2bacbb1483d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:45.830732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cffcc28402548458774bff31a89deec', 'request_id': 'req-eed23027-10ac-41f7-a4a8-c2bacbb1483d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:45.865821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] The token cache is being invalidate because project 4cffcc28402548458774bff31a89deec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:45.866218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:45.868325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eed23027-10ac-41f7-a4a8-c2bacbb1483d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:45.870641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 990/1977] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:45 2022] DELETE /identity/v3/projects/4cffcc28402548458774bff31a89deec => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:45.877954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a0fd-70d9-43d4-925d-3f10c13215ae None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.879389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a0fd-70d9-43d4-925d-3f10c13215ae None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.879547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a0fd-70d9-43d4-925d-3f10c13215ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.916425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0779a0fd-70d9-43d4-925d-3f10c13215ae None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:45.918513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0779a0fd-70d9-43d4-925d-3f10c13215ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:45.931938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf70e7-002c-4623-a98c-0a33f66c2a7c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:45.938993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:45.939699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.939942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.940219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.941924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:45.943669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 991/1978] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:45.944553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:45.949024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:45.949674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:45.950365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:45.996642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 988/1979] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/users => generated 291 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:45.998375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:46.000942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:46.003091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.007325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:46.025275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f44210-cc4d-4267-ac3b-37ef397b87da admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.031270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.032139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.032572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.033014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.034027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.036079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.048113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 992/1980] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:46.049731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.059373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 989/1981] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:36:46.070664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.075256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bdc7fb-e167-412a-b4a1-220baf944f14 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.085529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f1b99-58b6-42b9-a46c-ab397edb9af0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.093255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.094163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.094438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.094742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] PATH_INFO: `/v3/projects/2f9c0e5de599472ebe851608b0ee027c/users/c62b9b874fbd4758b30ed9df3b6f4a03/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.099640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.100060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.100610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.102491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.103778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:46.104306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.110648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.122307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f9c0e5de599472ebe851608b0ee027c, user_id=c62b9b874fbd4758b30ed9df3b6f4a03, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.126334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.176161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 990/1982] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:46 2022] PUT /identity/v3/projects/2f9c0e5de599472ebe851608b0ee027c/users/c62b9b874fbd4758b30ed9df3b6f4a03/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:46.182664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.186248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 993/1983] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:46.194860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.221615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fc234f-0164-47f1-bd08-8134fa7c2d2c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.342802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:46.459709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28d1be3-32f4-4426-8b6f-f4654202bf35 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.463128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.468040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.468513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.468982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.470258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.475938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.483596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.484547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.484998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.485450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.487200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.489429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.542304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 991/1984] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:46.546442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.563447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 994/1985] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:46 2022] POST /identity/v3/projects => generated 385 bytes in 372 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:46.577103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.620429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e210c564-ef2c-4cc3-9b2f-c35b34b44535 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.715772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:46.765109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.766496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.766682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f185d61-28ad-4b12-85fe-697d78ac033f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.767006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.767471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] PATH_INFO: `/v3/projects/2f9c0e5de599472ebe851608b0ee027c/users/c62b9b874fbd4758b30ed9df3b6f4a03/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.773164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f9c0e5de599472ebe851608b0ee027c, user_id=c62b9b874fbd4758b30ed9df3b6f4a03, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.778380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dffe9b-8eff-40cb-b05e-b9f472fca700 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.782652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.783728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.784004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.784531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.786571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.790724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.807157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 992/1986] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:46 2022] PUT /identity/v3/projects/2f9c0e5de599472ebe851608b0ee027c/users/c62b9b874fbd4758b30ed9df3b6f4a03/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:46.819080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.819429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.819640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.834574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 995/1987] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:46 2022] POST /identity/v3/users => generated 293 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:46.839351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:46.854429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4e9f8a-d5a9-4848-b2c2-b5ec7b542348 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:46.893395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] MFA Rules not processed for user `c62b9b874fbd4758b30ed9df3b6f4a03`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:46.895121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:46.934223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:46.934842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:46.935118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:46.935389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:46.936242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:46.938627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:46.968799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 996/1988] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:36:46.973486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.001794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e1e34b-5a9d-4bb3-bc92-d696f3d460fc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.015275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.016261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.019208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.020499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.022321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.043237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.056887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 997/1989] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:47.065177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.090359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9715a3-1d87-452c-8b79-a8bda4910f75 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.093848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 993/1990] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:46 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:47.104883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.106129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.107196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.107477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.107745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] PATH_INFO: `/v3/projects/6737f3cea6704da18efa353577a2c311/users/051a7ce183a941ef830ae9f730b14c22/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.108526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:47.132680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] RBAC: Authorizing `identity:create_grant(project_id=6737f3cea6704da18efa353577a2c311, user_id=051a7ce183a941ef830ae9f730b14c22, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.136009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc4a366-90a8-4c09-9b6b-7137a1efbefa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.173550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 998/1991] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:47 2022] PUT /identity/v3/projects/6737f3cea6704da18efa353577a2c311/users/051a7ce183a941ef830ae9f730b14c22/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:47.177644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.177826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.178211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.211544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f70c97e-7cda-493d-93c9-11860bf925f7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.222581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.231371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] MFA Rules not processed for user `051a7ce183a941ef830ae9f730b14c22`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:47.232772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:47.350240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.352966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.354079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.355617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.391015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 999/1992] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:47.403510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.439060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:47.556786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab147b5-ded1-4d43-9501-d8c3d35b7730 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.569901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.574689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.575766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.576807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.577242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.577938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.580576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.582676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.667230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 994/1993] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 561 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:36:47.685272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1000/1994] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:47 2022] POST /identity/v3/projects => generated 385 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:47.691456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.715357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b300b-f957-4b02-8251-322c06dc5a9d service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.746825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.752739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:47.887482 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.893858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.893858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.893858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.893858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:47.893858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:47.947389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 995/1995] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:47 2022] POST /identity/v3/users => generated 293 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:47.960734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96815c-3aaa-49e3-9912-a8e2cfd7d145 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:47.967663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:47.968594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:47.968958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:47.969307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:47.979830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:47.980998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:48.019339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d1cd-7968-4223-841b-aad532902202 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.037975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.038775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.039153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.039494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.040284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.042066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.092567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 996/1996] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:48.096089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.124299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93440486-b782-418d-8eeb-2107675373c3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.131102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.131876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.132258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.132638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.133477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.134456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.135359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a018a8b3-bb7e-454c-a77e-a36b234afcd3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.136821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.160648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1001/1997] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:48.195627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 997/1998] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:48 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:48.196876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.215877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c65ad8-9348-4166-ae6a-85ae882f73c3 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.230664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.231388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.231668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.231975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] PATH_INFO: `/v3/projects/648f55f9f8ca4a9f9259ee6b69989e49/users/8e9b8173f9ee4ab3b36d78b4b67bb879/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.272121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] RBAC: Authorizing `identity:create_grant(project_id=648f55f9f8ca4a9f9259ee6b69989e49, user_id=8e9b8173f9ee4ab3b36d78b4b67bb879, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.275687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.295609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1002/1999] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:48 2022] PUT /identity/v3/projects/648f55f9f8ca4a9f9259ee6b69989e49/users/8e9b8173f9ee4ab3b36d78b4b67bb879/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:48.311619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.312156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.312458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.401172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] MFA Rules not processed for user `8e9b8173f9ee4ab3b36d78b4b67bb879`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:48.403889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:48.452391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.543849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 998/2000] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:48.548039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.686446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:48.732841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7b9678-8d03-4dfd-88e3-de3e92bbf574 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.739689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.740454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.740843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.741243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.742091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.744386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:48.794667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1003/2001] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:48 2022] POST /identity/v3/projects => generated 385 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:48.802919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:48.828814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade7e9b1-6169-4aab-93e2-e0100ac03e5d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:48.982465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:48.982826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:48.983139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:48.983511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:48.984437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:48.987034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.040936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 999/2002] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:48 2022] POST /identity/v3/users => generated 299 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:49.044706 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.056833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95338d30-15f4-448c-ae03-651a97f22bfc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.074495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.076020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.076414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.076864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.078067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.081501 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.112217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1004/2003] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:36:49.116332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.146165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0d17f5-657b-4948-a601-3cba2128552c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.161316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.162008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.162293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.162542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] PATH_INFO: `/v3/projects/e44f4ff446c04054b0589797c43d60db/users/511a8cedecf04af585e616080d678b2e/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.180810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44f4ff446c04054b0589797c43d60db, user_id=511a8cedecf04af585e616080d678b2e, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.184002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.202657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1000/2004] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:49 2022] PUT /identity/v3/projects/e44f4ff446c04054b0589797c43d60db/users/511a8cedecf04af585e616080d678b2e/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:49.207497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.221047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f68e-4c62-478b-b775-1cf1e5eb293a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.365411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.366119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.366825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.367087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.367802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.369937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60394a83-2ba4-4533-936d-4ee5c7d1084b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.411125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1005/2005] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:49.413238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.444788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2ea0-5244-431e-908b-8702f8122f1e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.455321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.457441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.458494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.459211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] PATH_INFO: `/v3/projects/e44f4ff446c04054b0589797c43d60db/users/511a8cedecf04af585e616080d678b2e/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.462723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44f4ff446c04054b0589797c43d60db, user_id=511a8cedecf04af585e616080d678b2e, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.469148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.503776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1001/2006] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:49 2022] PUT /identity/v3/projects/e44f4ff446c04054b0589797c43d60db/users/511a8cedecf04af585e616080d678b2e/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:49.508698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.509551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.510228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.583888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] MFA Rules not processed for user `511a8cedecf04af585e616080d678b2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:49.587661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:49.721333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1006/2007] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:49 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:49.734792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:49.845231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb120a93-ce90-4436-86cb-9ae9878a6403 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:49.849854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:49.850516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:49.850907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:49.851181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:49.885397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:49.946347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:49.948410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:49.963315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1002/2008] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:49 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:52.407716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:52.428121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee36bca-1d29-403f-a953-36c132e9c726 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:52.436596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:52.437626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:52.438145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:52.438677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:52.543564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:52.629787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:52.633653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d9c9bc-43ad-4255-a185-f9e11bf2bc2f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:52.688450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1007/2009] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:53.596422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:53.779058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4bcdd-3e7a-4113-81df-9fa2e6b5935a service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:53.785774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:53.786654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:53.787089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:53.787483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] PATH_INFO: `/v3/users/b6698bed548945b081d1c101b7803263` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:53.789067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6698bed548945b081d1c101b7803263)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:53.793541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:53.935336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6698bed548945b081d1c101b7803263'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:53.945049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6698bed548945b081d1c101b7803263'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:53.990293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6698bed548945b081d1c101b7803263'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:53.992049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6698bed548945b081d1c101b7803263'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:53.992947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6698bed548945b081d1c101b7803263'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.025683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1003/2010] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:53 2022] DELETE /identity/v3/users/b6698bed548945b081d1c101b7803263 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:54.042442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28aa95-e1bb-4f7f-a3da-47c24be9280f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:54.043827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28aa95-e1bb-4f7f-a3da-47c24be9280f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:54.045545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28aa95-e1bb-4f7f-a3da-47c24be9280f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:54.171747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ad28aa95-e1bb-4f7f-a3da-47c24be9280f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:54.177600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ad28aa95-e1bb-4f7f-a3da-47c24be9280f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.213100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:54.214916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.464008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:54.479890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1008/2011] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:54.484120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:54.490296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:54.491786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:54.493601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1009/2012] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:36:54 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:36:54.494726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:54.500619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.507182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805b1c3-966d-46a3-9d32-45a232953194 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:54.525118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:54.526252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:54.526814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:54.527172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:54.529988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.550447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ecfb1-580e-44e5-94fc-896f801773f6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:54.566181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:54.568962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:54.569583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:54.570080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] PATH_INFO: `/v3/users/d002a844a71649d884b4ca870eb9d387` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:54.576263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d002a844a71649d884b4ca870eb9d387)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:54.579041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:54.674241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:54.676301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:54.682126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd002a844a71649d884b4ca870eb9d387'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.689285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1004/2013] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:54 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:54.690172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:54.692544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.712152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd002a844a71649d884b4ca870eb9d387'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.746533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd002a844a71649d884b4ca870eb9d387'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.749016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd002a844a71649d884b4ca870eb9d387'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.765738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e3c300d-797c-411d-8d66-3211b14e541b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd002a844a71649d884b4ca870eb9d387'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:54.779405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1010/2014] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:54 2022] DELETE /identity/v3/users/d002a844a71649d884b4ca870eb9d387 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:54.780725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968f18-08ea-4cd8-aa90-3869f88377b2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:54.781006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968f18-08ea-4cd8-aa90-3869f88377b2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:54.781273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968f18-08ea-4cd8-aa90-3869f88377b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:54.834378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-54968f18-08ea-4cd8-aa90-3869f88377b2 None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:54.841305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-54968f18-08ea-4cd8-aa90-3869f88377b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:54.856811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8236642-1489-40bf-9b2c-6faa3555bdf2 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:54.969065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:55.055883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1011/2015] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:36:54 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:55.059115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.059459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.060419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.069942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.070600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.070883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.071166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.093120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.152419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] MFA Rules not processed for user `ab4b3c2b3f4b4978b0eb472f9c2736b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:55.152419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.272171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:55.277211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.283547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.299141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1012/2016] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:55.303174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.303174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.303354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1005/2017] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:55.306794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.309499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.328819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be98d3b-14df-4687-a713-1a70009afc3e service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.336015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.336466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e78507-2269-4ff9-a352-680e5bb88a7c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.336889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.337155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.337437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] PATH_INFO: `/v3/projects/3f34eba992024f6881d1292c9f7cfd48` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.338740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f34eba992024f6881d1292c9f7cfd48)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.340866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.344100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.347902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.348334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.348621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.360237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.363408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.378859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1013/2018] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:55.389680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.391566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.417859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d6445-db69-429f-b547-8279b654b766 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.429956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.430937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.431220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.431493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.432214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.434846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.454460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f34eba992024f6881d1292c9f7cfd48', 'request_id': 'req-993e111b-35bc-4d21-b380-671a8739c848'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:55.469590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f34eba992024f6881d1292c9f7cfd48', 'request_id': 'req-993e111b-35bc-4d21-b380-671a8739c848'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:55.486759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1014/2019] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/projects => generated 379 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:55.492578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.495062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f34eba992024f6881d1292c9f7cfd48', 'request_id': 'req-993e111b-35bc-4d21-b380-671a8739c848'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:55.514020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] The token cache is being invalidate because project 3f34eba992024f6881d1292c9f7cfd48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:55.514472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:55.515638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-993e111b-35bc-4d21-b380-671a8739c848 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:55.523232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1006/2020] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:55 2022] DELETE /identity/v3/projects/3f34eba992024f6881d1292c9f7cfd48 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:55.529898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39462f7-fb2a-4e6b-81b4-dc7ed1b0ae6a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.530204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39462f7-fb2a-4e6b-81b4-dc7ed1b0ae6a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.530603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39462f7-fb2a-4e6b-81b4-dc7ed1b0ae6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.581139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b39462f7-fb2a-4e6b-81b4-dc7ed1b0ae6a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:55.583177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b39462f7-fb2a-4e6b-81b4-dc7ed1b0ae6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.588713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf4457a-6f6e-4b63-b096-bb71a3907d4b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.598592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.598879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1007/2021] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:55.603215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.603661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.604583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.605506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.606171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.606462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.606778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.608768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.651463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:55.653171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.656847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1015/2022] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/users => generated 296 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:55.667171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.669705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:55.682852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1008/2023] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:55.685494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.689108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6f91ba-89da-4ab6-83f8-6dd637ab800c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.695356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.696371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.697050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.697465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.698271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.698885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b61c880-959d-4313-b9a4-46aa25ffe936 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.700377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.705598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.707025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.707427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.707818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.709117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:55.709253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.711732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.721420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1016/2024] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:55 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:55.725832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.733182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1009/2025] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:55.738410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f77302-91d8-4db2-abf9-41529281f2e2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.739733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.743691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.744396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.744661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.744930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] PATH_INFO: `/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76/users/80c68cf98eea4ceaae245f70ad7a1b8b/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.757314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:55.762043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f496837c-fca6-44ec-b40e-90c350d051a5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.766508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9f8f3f4a8dc45919e6bc8a78b745a76, user_id=80c68cf98eea4ceaae245f70ad7a1b8b, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.770131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.773374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.773580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.773849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.774128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.774842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.776810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:55.794878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1017/2026] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:55 2022] PUT /identity/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76/users/80c68cf98eea4ceaae245f70ad7a1b8b/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:36:55.797118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.839148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1010/2027] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/projects => generated 375 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:55.844473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:55.919020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e5985f-380c-4744-ae9a-5d2869ace044 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.959045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94bb979-0b8a-49d9-9ca9-12caeb423973 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:55.964882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:55.966002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:55.966118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:55.966308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:55.967350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:55.969469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.001285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1011/2028] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:55 2022] POST /identity/v3/users => generated 288 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:56.002196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.004287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:56.017596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef72481-d86b-45bc-bbce-c78664105f47 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.019460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.020329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.020614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.020967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.021923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.023855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.024621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.025380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.025732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.026073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] PATH_INFO: `/v3/projects/3c243001532b4a398a8ce8578522baf4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.027418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c243001532b4a398a8ce8578522baf4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.029535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.052202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.054710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1018/2029] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:55 2022] GET /identity/v3/roles => generated 1420 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:56.056197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.072668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e6ff6-0dd7-4bbd-977f-2a4adaf06721 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.078334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.078773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.079056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.079327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.080092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.082351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.110483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1019/2030] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:56.110483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.118376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c243001532b4a398a8ce8578522baf4', 'request_id': 'req-c7cf67ca-6650-420e-9f7b-94b7932eba95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.127330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c972e666-6d43-40bf-8cc1-bb2e858f1281 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.133958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c243001532b4a398a8ce8578522baf4', 'request_id': 'req-c7cf67ca-6650-420e-9f7b-94b7932eba95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.170578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c243001532b4a398a8ce8578522baf4', 'request_id': 'req-c7cf67ca-6650-420e-9f7b-94b7932eba95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.203778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] The token cache is being invalidate because project 3c243001532b4a398a8ce8578522baf4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:56.206465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.211550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.216481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1012/2031] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:56 2022] DELETE /identity/v3/projects/3c243001532b4a398a8ce8578522baf4 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:56.221383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.229897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:56.321811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.322513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.322853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.323205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] PATH_INFO: `/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76/users/80c68cf98eea4ceaae245f70ad7a1b8b/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.325249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9f8f3f4a8dc45919e6bc8a78b745a76, user_id=80c68cf98eea4ceaae245f70ad7a1b8b, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.329217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.359541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1020/2032] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:56 2022] PUT /identity/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76/users/80c68cf98eea4ceaae245f70ad7a1b8b/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:56.367053 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.370638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:56.406506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:56.418166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cf67ca-6650-420e-9f7b-94b7932eba95 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.553649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5703d55-a4a5-4a19-9382-8f14c04941ea admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.569100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.569935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.570625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.571065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] PATH_INFO: `/v3/users/c62b9b874fbd4758b30ed9df3b6f4a03` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.573067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c62b9b874fbd4758b30ed9df3b6f4a03)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.575720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.604660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.604660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.604660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.604660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.605336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.605336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.644559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1013/2033] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:56 2022] GET /identity/v3/roles => generated 1420 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:56.646057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:56.683992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d53183-10f3-4401-bcd2-4dbc367aab63 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:56.686710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c62b9b874fbd4758b30ed9df3b6f4a03'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.701016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:56.703518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.703868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.704449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] PATH_INFO: `/v3/projects/5d21789971964770bdf417547cd19840/users/78634b34107d49cdbf0126014bf8f6f5/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.711767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c62b9b874fbd4758b30ed9df3b6f4a03'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.764614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d21789971964770bdf417547cd19840, user_id=78634b34107d49cdbf0126014bf8f6f5, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:56.769284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c62b9b874fbd4758b30ed9df3b6f4a03'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.771905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c62b9b874fbd4758b30ed9df3b6f4a03'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.772293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:56.790025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7725a1b9-970a-477d-98fe-c3d80a80afc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c62b9b874fbd4758b30ed9df3b6f4a03'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:56.811655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1021/2034] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:56 2022] DELETE /identity/v3/users/c62b9b874fbd4758b30ed9df3b6f4a03 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:56.816443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:56.816781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:56.816991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:56.829159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1014/2035] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:56 2022] PUT /identity/v3/projects/5d21789971964770bdf417547cd19840/users/78634b34107d49cdbf0126014bf8f6f5/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:56.900221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:56.902180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:57.053588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1022/2036] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:36:57.085686 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.088666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:57.169289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.278474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65552ca8-0229-4d0e-9cfd-ebb5d00a9749 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.415837 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.416819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.417310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.417829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.442502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.444429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20babe4c-ca37-41d2-91eb-579528a6c764 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.466814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1015/2037] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:36:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:36:57.503843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:57.505515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:57.520649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a662fb80-0e65-4119-902d-abb4afdf555d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:57.527199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:57.528102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.528403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.528797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] PATH_INFO: `/v3/projects/2f9c0e5de599472ebe851608b0ee027c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.533631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f9c0e5de599472ebe851608b0ee027c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:57.536097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:57.595332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:57.686203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f9c0e5de599472ebe851608b0ee027c', 'request_id': 'req-181e84e4-36f5-482f-9e7d-b5b5752102a3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:57.699203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f9c0e5de599472ebe851608b0ee027c', 'request_id': 'req-181e84e4-36f5-482f-9e7d-b5b5752102a3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:57.724593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f9c0e5de599472ebe851608b0ee027c', 'request_id': 'req-181e84e4-36f5-482f-9e7d-b5b5752102a3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:57.756972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] The token cache is being invalidate because project 2f9c0e5de599472ebe851608b0ee027c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:36:57.757931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:57.758224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-181e84e4-36f5-482f-9e7d-b5b5752102a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:36:57.761468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1023/2038] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:36:57 2022] DELETE /identity/v3/projects/2f9c0e5de599472ebe851608b0ee027c => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:57.766906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.767838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.768376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:57.798985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:57.800777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:57.968894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1016/2039] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:57.979614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:57.980510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:57.981089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.073901 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:58.075317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:58.112039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1024/2040] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:58.114307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.132053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b3f25e-adce-44f2-87e5-600473afa834 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.139803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.140552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.141166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.141449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.142220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:36:58.143217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.147454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.200661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.200921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1017/2041] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:36:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:36:58.212713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a75080-285d-4384-b4fc-b91404d8cfa0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.229531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.230325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.230717 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.231055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.231813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.234396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.314665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1025/2042] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:58 2022] POST /identity/v3/projects => generated 381 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:58.315929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.344958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81cf51-5f8d-45e7-afd3-1e6b72dbdf22 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.572193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.573107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.573401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.573703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.574692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.580662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.636196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:36:58.672809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1018/2043] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:58 2022] POST /identity/v3/users => generated 291 bytes in 361 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:58.674578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.707443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b60e1e-0dee-4b66-bbae-85d5b743c742 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.728657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.729805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.730831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.731227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.733470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.736710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.779443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1026/2044] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:58.781551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.812211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e31c95-1893-4d2d-9f29-f9127922e51b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.828773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.829437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.829716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.830329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.831088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.832978 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ceda0-1bce-4387-9e2b-0dad35faeda4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.867001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1019/2045] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:58.871319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:58.890183 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70123136-a0a1-4b60-af9d-dc9fb9766647 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:58.908982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:58.909448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.909877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.910734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] PATH_INFO: `/v3/projects/ec4f0ed94b974ff8baeb2b6c6f7f7100/users/7eb62d0ffc2e4b8b9b7d0937ddaaedef/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:58.953409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec4f0ed94b974ff8baeb2b6c6f7f7100, user_id=7eb62d0ffc2e4b8b9b7d0937ddaaedef, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:58.959298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:58.984753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1027/2046] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:58 2022] PUT /identity/v3/projects/ec4f0ed94b974ff8baeb2b6c6f7f7100/users/7eb62d0ffc2e4b8b9b7d0937ddaaedef/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:58.996265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:58.996538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:58.996821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.056684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] MFA Rules not processed for user `7eb62d0ffc2e4b8b9b7d0937ddaaedef`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:36:59.057989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:59.203183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1020/2047] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:36:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:36:59.208430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.362830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba67ee7-2be7-4e3b-8ea1-4ba7650e717f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:59.371423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:59.372992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:59.374061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:59.374680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.375483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:59.385909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:59.478636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1028/2048] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:36:59 2022] POST /identity/v3/projects => generated 383 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:36:59.480886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.505137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8a103-862f-4da4-b00c-7e983bbaa617 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:59.687179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:59.689525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:59.690132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:59.694657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.695642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:59.701685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:59.745996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1021/2049] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:36:59 2022] POST /identity/v3/users => generated 298 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:36:59.749719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.762090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25e5fb6-8715-4c9b-92ad-e331eed6d15d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:59.775940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:59.777413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:59.778977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:59.779399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.783139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:59.785627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:59.795110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.798161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:36:59.807751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1029/2050] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:59 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:36:59.814210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.850941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c44e0dc-1d8e-4264-bb02-35caa5929a90 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:59.860324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc4008-a229-48bd-8239-80927856b41d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:36:59.869121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:59.870021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:59.870304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:59.872055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] PATH_INFO: `/v3/users/051a7ce183a941ef830ae9f730b14c22` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.879607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:36:59.881484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:36:59.881801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:36:59.882818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] PATH_INFO: `/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a/users/5164944deee24387a73e8cb758ca6697/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:36:59.895003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] RBAC: Authorizing `identity:delete_user(user_id=051a7ce183a941ef830ae9f730b14c22)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:59.898195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:59.904799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5dbba5e55a444bb9be889d01e8b31c6a, user_id=5164944deee24387a73e8cb758ca6697, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:36:59.908573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:36:59.938406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1030/2051] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:36:59 2022] PUT /identity/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a/users/5164944deee24387a73e8cb758ca6697/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:36:59.943972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:36:59.957951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.003693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '051a7ce183a941ef830ae9f730b14c22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:00.024798 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '051a7ce183a941ef830ae9f730b14c22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:00.045873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.076727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '051a7ce183a941ef830ae9f730b14c22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:00.077900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '051a7ce183a941ef830ae9f730b14c22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:00.078956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eee69492-602f-40ad-9d2f-92a6cce48784 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '051a7ce183a941ef830ae9f730b14c22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:00.093147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b01e8-451c-42fd-9bed-a7d7e82227f9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:00.106371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:00.108360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.110218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.111025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.112774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:00.117065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:00.121559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1022/2052] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:36:59 2022] DELETE /identity/v3/users/051a7ce183a941ef830ae9f730b14c22 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:00.128881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.129194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.129455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.139095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1031/2053] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:36:59 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:37:00.143473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:00.146989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.173981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:00.176031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.176585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7817b5a-5abf-4071-a99b-44d2101dcfd6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:00.307350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:00.309836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.310633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.311285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] PATH_INFO: `/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a/users/5164944deee24387a73e8cb758ca6697/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.316692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5dbba5e55a444bb9be889d01e8b31c6a, user_id=5164944deee24387a73e8cb758ca6697, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:00.327169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f5551-58ed-48f9-96be-8c5d68ed1e16 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:00.376664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1032/2054] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:00 2022] PUT /identity/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a/users/5164944deee24387a73e8cb758ca6697/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:00.380718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1023/2055] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:00.392997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.394611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.394611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.401246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:00.403164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.442636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] MFA Rules not processed for user `5164944deee24387a73e8cb758ca6697`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:00.443896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.560896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49c17e8-4585-4727-974a-30ad9d15af5f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:00.576657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1033/2056] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:00 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:37:00.590202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:00.593424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:00.640407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.694024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:00.694024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.694024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.694024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.777158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5224f963-f87b-46c1-ab0e-6efd66195d43 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:00.796522 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:00.797720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:00.798222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:00.800296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:00.800796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:00.835933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:00.837694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:00.848337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:00.852139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:00.856211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1024/2057] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:00.880207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1034/2058] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:00 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:01.077512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:01.092486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d907557-ffcb-42f6-b362-e1f1bef077d2 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:01.103744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:01.105004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:01.105493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:01.105660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:01.114222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:01.330318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:01.332081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:01.350318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1025/2059] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:01 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:02.596806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.599439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:02.622354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513bcabb-e028-4174-908f-65af6cd69625 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.637503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.638688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.639046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.639418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] PATH_INFO: `/v3/projects/6737f3cea6704da18efa353577a2c311` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.641660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6737f3cea6704da18efa353577a2c311)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.645507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:02.730987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6737f3cea6704da18efa353577a2c311', 'request_id': 'req-a971168b-7ed1-42f6-9ebc-be826623fa28'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:02.740083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6737f3cea6704da18efa353577a2c311', 'request_id': 'req-a971168b-7ed1-42f6-9ebc-be826623fa28'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:02.755796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6737f3cea6704da18efa353577a2c311', 'request_id': 'req-a971168b-7ed1-42f6-9ebc-be826623fa28'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:02.790976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] The token cache is being invalidate because project 6737f3cea6704da18efa353577a2c311 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:02.791607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:02.793598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:02.804588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1035/2060] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:02 2022] DELETE /identity/v3/projects/6737f3cea6704da18efa353577a2c311 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:02.805503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:02.820568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b122200-be7c-45b2-9866-38b404552c8a service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:02.962114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:02.964027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:02.964215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:02.964720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] PATH_INFO: `/v3/users/8e9b8173f9ee4ab3b36d78b4b67bb879` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:02.973202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e9b8173f9ee4ab3b36d78b4b67bb879)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:02.975962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:03.072383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e9b8173f9ee4ab3b36d78b4b67bb879'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.105060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e9b8173f9ee4ab3b36d78b4b67bb879'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.137494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e9b8173f9ee4ab3b36d78b4b67bb879'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.139138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e9b8173f9ee4ab3b36d78b4b67bb879'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.139803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e9b8173f9ee4ab3b36d78b4b67bb879'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.197979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1026/2061] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:02 2022] DELETE /identity/v3/users/8e9b8173f9ee4ab3b36d78b4b67bb879 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:03.682304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:03.684416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:03.715764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a971168b-7ed1-42f6-9ebc-be826623fa28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:03.819990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:03.821406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:03.822341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:03.822955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] PATH_INFO: `/v3/projects/648f55f9f8ca4a9f9259ee6b69989e49` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:03.825481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=648f55f9f8ca4a9f9259ee6b69989e49)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:03.828749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:03.924102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:03.933023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '648f55f9f8ca4a9f9259ee6b69989e49', 'request_id': 'req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.948947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '648f55f9f8ca4a9f9259ee6b69989e49', 'request_id': 'req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.961674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '648f55f9f8ca4a9f9259ee6b69989e49', 'request_id': 'req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.985758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] The token cache is being invalidate because project 648f55f9f8ca4a9f9259ee6b69989e49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:03.986306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.987483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:03.989835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1036/2062] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:03 2022] DELETE /identity/v3/projects/648f55f9f8ca4a9f9259ee6b69989e49 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:03.995404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:03.998013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:04.011903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:04.020568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c92-c73b-4071-8772-655ae0bd6b91 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:04.136614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:04.137438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:04.137815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:04.138192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] PATH_INFO: `/v3/users/511a8cedecf04af585e616080d678b2e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:04.146656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] RBAC: Authorizing `identity:delete_user(user_id=511a8cedecf04af585e616080d678b2e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:04.146656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:04.252021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '511a8cedecf04af585e616080d678b2e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:04.263920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '511a8cedecf04af585e616080d678b2e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:04.291760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '511a8cedecf04af585e616080d678b2e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:04.291760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '511a8cedecf04af585e616080d678b2e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:04.292761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bdf36fc0-1ec8-4e89-a20f-9489aaee8119 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '511a8cedecf04af585e616080d678b2e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:04.324365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1027/2063] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:03 2022] DELETE /identity/v3/users/511a8cedecf04af585e616080d678b2e => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:04.677299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:04.679669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:04.727779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d52a9ca-3431-4bcd-99ad-62d3d26eff40 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:04.853542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:04.853542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:04.853973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:04.853973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] PATH_INFO: `/v3/users/7eb62d0ffc2e4b8b9b7d0937ddaaedef` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:04.857645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7eb62d0ffc2e4b8b9b7d0937ddaaedef)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:04.868804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:04.999212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7eb62d0ffc2e4b8b9b7d0937ddaaedef'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.013104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7eb62d0ffc2e4b8b9b7d0937ddaaedef'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.076615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7eb62d0ffc2e4b8b9b7d0937ddaaedef'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.077141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7eb62d0ffc2e4b8b9b7d0937ddaaedef'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.087002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7eb62d0ffc2e4b8b9b7d0937ddaaedef'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.100641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1037/2064] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:04 2022] DELETE /identity/v3/users/7eb62d0ffc2e4b8b9b7d0937ddaaedef => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:05.104952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.105810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.106263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.150499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:05.155139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:05.295612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.297607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:05.387001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1028/2065] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:05.465884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:05.469402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:05.507940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.546348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f4e6e9-3fc3-4028-a8f8-2428307d9308 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.559775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.563767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.564335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.565926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] PATH_INFO: `/v3/projects/e44f4ff446c04054b0589797c43d60db` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.567839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:05.568803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e44f4ff446c04054b0589797c43d60db)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:05.605961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:05.689742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f7c8d4-49e8-4bff-813d-8efe004df841 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:05.697853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:05.699862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.700332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.700848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.745829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:05.747479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213014e8-3ea6-4604-aa6b-385705297d47 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:05.766592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1029/2066] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:05.786680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e44f4ff446c04054b0589797c43d60db', 'request_id': 'req-bb0accaf-a84f-4d55-b76f-c0046466c992'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.803686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e44f4ff446c04054b0589797c43d60db', 'request_id': 'req-bb0accaf-a84f-4d55-b76f-c0046466c992'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.823724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e44f4ff446c04054b0589797c43d60db', 'request_id': 'req-bb0accaf-a84f-4d55-b76f-c0046466c992'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.854009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] The token cache is being invalidate because project e44f4ff446c04054b0589797c43d60db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:05.854009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.856912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb0accaf-a84f-4d55-b76f-c0046466c992 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:05.859414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1038/2067] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:05 2022] DELETE /identity/v3/projects/e44f4ff446c04054b0589797c43d60db => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:05.868997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:05.871783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:05.872816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:05.900344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:05.901932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:06.004188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1030/2068] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:06.005778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.006074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.007257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.050642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:06.054454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:06.079958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1039/2069] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:06.083672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.110106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798f75cd-0097-405b-b6f0-23efcfc06ea3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.147102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.156293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.157781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.158082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.159435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:06.170763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.172574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.223856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1031/2070] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:06.225779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.253927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ba20d0-63c0-46f1-8068-7b3ce77063fd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.286190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.287260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.287735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.288203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.295329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.299206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.433855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1040/2071] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:06 2022] POST /identity/v3/projects => generated 373 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:06.438594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.459246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26d5312-d693-4834-a267-37be0752427c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.617084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.617916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.618351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.618812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.619661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.621871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.659476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.661358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:06.668079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1032/2072] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:06 2022] POST /identity/v3/users => generated 287 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:06.671959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.693850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ceda3-523a-4fa3-aaf0-761032842189 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.695289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ef01fe-c655-4341-92ab-273390b5390a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.709485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.710143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.710467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.710805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] PATH_INFO: `/v3/projects/ec4f0ed94b974ff8baeb2b6c6f7f7100` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.712228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec4f0ed94b974ff8baeb2b6c6f7f7100)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.714340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.715153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.717071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.717362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.717637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.718467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.725846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.762996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1033/2073] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:37:06.763826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.781938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b5ca1a-aa75-4c22-b1f1-7de2278407db admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.796488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.797119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.797423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.797654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.798306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.800160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.830963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1034/2074] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:06.835566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:06.863230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771b5f61-e25f-4714-8db2-fd2ca6a4594e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:06.876222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:06.876222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.876222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.876551 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] PATH_INFO: `/v3/projects/f758e7615232486a93a915760633b702/users/76b2078f1f64440e9d8faca3bcf98d52/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.914445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f758e7615232486a93a915760633b702, user_id=76b2078f1f64440e9d8faca3bcf98d52, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:06.917514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bb3271-5d5e-4986-b775-d7dc6fa96041 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:06.924144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:06.928386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec4f0ed94b974ff8baeb2b6c6f7f7100', 'request_id': 'req-61b496c8-2a54-450e-b4f1-4df848657639'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:06.947550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1035/2075] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:06 2022] PUT /identity/v3/projects/f758e7615232486a93a915760633b702/users/76b2078f1f64440e9d8faca3bcf98d52/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:06.952329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec4f0ed94b974ff8baeb2b6c6f7f7100', 'request_id': 'req-61b496c8-2a54-450e-b4f1-4df848657639'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:06.952808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:06.953278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:06.957629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:06.980255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec4f0ed94b974ff8baeb2b6c6f7f7100', 'request_id': 'req-61b496c8-2a54-450e-b4f1-4df848657639'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.005883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] The token cache is being invalidate because project ec4f0ed94b974ff8baeb2b6c6f7f7100 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:07.007201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.008264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.012188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1041/2076] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:06 2022] DELETE /identity/v3/projects/ec4f0ed94b974ff8baeb2b6c6f7f7100 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:07.016509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:07.020794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:07.050039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:07.051784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:07.236661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:07.342724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1036/2077] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:07.368140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:07.370331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:07.370471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b496c8-2a54-450e-b4f1-4df848657639 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:07.383580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:07.384285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:07.384568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:07.384840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] PATH_INFO: `/v3/users/5164944deee24387a73e8cb758ca6697` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:07.386476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=5164944deee24387a73e8cb758ca6697)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:07.391655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:07.547801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5164944deee24387a73e8cb758ca6697'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.569324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5164944deee24387a73e8cb758ca6697'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.577117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dbb5b5-2874-427f-a362-7ab2d95a47f8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:07.588174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:07.589501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:07.590021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:07.590597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:07.607133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5164944deee24387a73e8cb758ca6697'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.610952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:07.611440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5164944deee24387a73e8cb758ca6697'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.615350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:07.630791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1037/2078] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:07.635403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c67dd1fe-8a96-4111-b242-14f8a06097ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5164944deee24387a73e8cb758ca6697'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:07.648268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1042/2079] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:07 2022] DELETE /identity/v3/users/5164944deee24387a73e8cb758ca6697 => generated 0 bytes in 632 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:09.473316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:09.491734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919fb513-d613-472a-b55f-b7c34e361562 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:09.797614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:09.798261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:09.798599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:09.798830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] PATH_INFO: `/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:09.804915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5dbba5e55a444bb9be889d01e8b31c6a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:09.815658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.017549 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.038380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5dbba5e55a444bb9be889d01e8b31c6a', 'request_id': 'req-259fad36-a2cd-4123-8f0d-427e4c7c877d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.052223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5dbba5e55a444bb9be889d01e8b31c6a', 'request_id': 'req-259fad36-a2cd-4123-8f0d-427e4c7c877d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.084375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de362a9c-88d1-49e4-8d48-e43ceb2be20e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.085127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de362a9c-88d1-49e4-8d48-e43ceb2be20e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.085127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de362a9c-88d1-49e4-8d48-e43ceb2be20e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.088155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5dbba5e55a444bb9be889d01e8b31c6a', 'request_id': 'req-259fad36-a2cd-4123-8f0d-427e4c7c877d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.129745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] The token cache is being invalidate because project 5dbba5e55a444bb9be889d01e8b31c6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:10.129982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.131069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-259fad36-a2cd-4123-8f0d-427e4c7c877d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.139737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1038/2080] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:09 2022] DELETE /identity/v3/projects/5dbba5e55a444bb9be889d01e8b31c6a => generated 0 bytes in 663 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:10.146912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.146912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.146912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.172857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-de362a9c-88d1-49e4-8d48-e43ceb2be20e None None] MFA Rules not processed for user `78634b34107d49cdbf0126014bf8f6f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:10.174160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-de362a9c-88d1-49e4-8d48-e43ceb2be20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.243085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:10.249160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.459852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1039/2081] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:10.461729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.463655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.492950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e090fb-5df0-4573-9b39-bd7aeaf39792 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:10.519989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1043/2082] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 437 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:10.520547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:10.521226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.521542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.522199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] PATH_INFO: `/v3/users/75b28992f2034e40b5df5db29d321a46` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.530164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75b28992f2034e40b5df5db29d321a46)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:10.531415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:10.534464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:10.534957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:10.538469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:10.618048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:10.622166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.678154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1044/2083] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:10.685494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.693218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.758051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75b28992f2034e40b5df5db29d321a46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.786244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75b28992f2034e40b5df5db29d321a46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.864039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75b28992f2034e40b5df5db29d321a46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.865544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75b28992f2034e40b5df5db29d321a46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.869675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75b28992f2034e40b5df5db29d321a46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:10.905351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1040/2084] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:10 2022] DELETE /identity/v3/users/75b28992f2034e40b5df5db29d321a46 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:10.906411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:10.911389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:10.917533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:10.991757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd7276-50aa-4c0d-9017-734371e430d1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.021362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.059366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7959c4c-f380-4ad6-865f-bb160a929a74 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.065618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.066732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.066881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.067104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.069430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:11.069836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.071766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaa2a0e-8d00-4fce-9edf-0af27f127e39 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.093379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1041/2085] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:11.093379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.093379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.093379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.117198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:11.118797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.132162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1042/2086] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:11.133523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.151450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.152089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.152375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.152645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.155022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.167378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6364c-0f04-4eae-a00e-9284e48b6167 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.176119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.177740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.178307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.179565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.180797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.183235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.242127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1043/2087] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:11 2022] POST /identity/v3/projects => generated 369 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:11.244469 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.248218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.281834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.285245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.307355 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1045/2088] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 626 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:11.308488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.329266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb0a37d-5511-4786-841b-fb25d67f8112 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.334197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.334877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.335149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.335326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3c6e54-d8b0-451c-b900-16c966afd8dd service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.335458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.470617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.471838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.472052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.472372 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.472955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.480202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da86b33a-9678-4994-8ff8-7eea3642b6fc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.505585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.508512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.523544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1046/2089] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:11 2022] POST /identity/v3/users => generated 285 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:11.527231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.527777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.528643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.539972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1044/2090] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:11.540949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.547415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.555253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cf3fb7-9ad7-4827-802d-ffe1a0ab05b9 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.571454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.572281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.572601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.572889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.573886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.576847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.594707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] MFA Rules not processed for user `80c68cf98eea4ceaae245f70ad7a1b8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:11.597739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.605839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1045/2091] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:11.610075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.634008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a7fa5-96be-44b8-b070-e85f1135a13f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.655656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.656935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.656935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.657117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.657817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.660375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.685014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1046/2092] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:37:11.691303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.707617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c15884-08cf-4dbd-8c56-0fc0925aff5b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:11.717572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:11.718861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.719576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.720091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] PATH_INFO: `/v3/projects/8feb044ea9654cb9b3f0ab0c1442b6d1/users/ea8e0db2bb574e35bd78a9ef6650a44a/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.753258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8feb044ea9654cb9b3f0ab0c1442b6d1, user_id=ea8e0db2bb574e35bd78a9ef6650a44a, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:11.755340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1047/2093] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:11 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:11.774940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbefc819-9eef-443a-97b6-4f8441cc2462 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:11.776748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:11.782673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.795762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1047/2094] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:11 2022] PUT /identity/v3/projects/8feb044ea9654cb9b3f0ab0c1442b6d1/users/ea8e0db2bb574e35bd78a9ef6650a44a/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:11.809595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:11.812299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:11.812822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:11.867580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:11.902731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] MFA Rules not processed for user `ea8e0db2bb574e35bd78a9ef6650a44a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:11.907917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:11.999830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33932c9-c677-4ec1-8d7f-158ff3e7678c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:12.100068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1048/2095] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:12.110534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:12.116324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:12.128102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:12.129474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.130167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.130824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:12.135705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:12.210333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:12.278514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:12.281517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685e9c5d-db01-42fd-a140-c7a094c61c25 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:12.288711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:12.289544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.289924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.291382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:12.292865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:12.304434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:12.306333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013d3ae-80d2-4325-9afa-15a92869c1de service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:12.307407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1048/2096] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:12.323262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1049/2097] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:12.348215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:12.440246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.440996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.444176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:12.531125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] MFA Rules not processed for user `76b2078f1f64440e9d8faca3bcf98d52`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:12.534220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:12.578103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b63215-93d5-4610-abf7-4a579616546d service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:12.599580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:12.600783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.601235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.601592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] PATH_INFO: `/v3/users/ea8e0db2bb574e35bd78a9ef6650a44a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:12.606519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea8e0db2bb574e35bd78a9ef6650a44a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:12.619814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:12.807200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1050/2098] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:37:12.821221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:12.823036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:12.850722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4449d574-0553-427c-8cab-5fb8efe04212 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:12.852145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea8e0db2bb574e35bd78a9ef6650a44a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:12.862813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:12.865849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.866493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.867554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:12.875761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea8e0db2bb574e35bd78a9ef6650a44a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:12.891787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:12.895645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b189ae6-1300-47ec-813b-80312c95c940 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:12.919048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea8e0db2bb574e35bd78a9ef6650a44a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:12.920790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1051/2099] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:12.922303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea8e0db2bb574e35bd78a9ef6650a44a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:12.951239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea8e0db2bb574e35bd78a9ef6650a44a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:12.971896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1049/2100] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:12 2022] DELETE /identity/v3/users/ea8e0db2bb574e35bd78a9ef6650a44a => generated 0 bytes in 621 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:12.974280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:12.975511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:12.976091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:13.039861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:13.041946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:13.129553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:13.305465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1052/2101] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:13.343574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:13.348614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:13.521524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:13.560105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc4ab8-ce8a-4f14-897c-4e50a27b3078 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:13.575486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:13.579177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:13.580574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:13.582646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:13.606600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:13.609102 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1619dd7-9991-4c95-b6a4-468a72c84c98 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:13.629523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1050/2102] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:13.750093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:13.752105 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:13.789856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca07853-c74c-42b0-b3b2-f91ea5880471 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:13.813460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:13.814312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:13.820378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:13.820738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] PATH_INFO: `/v3/projects/3bd60df9238f4c1689138e1c28e74eca` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:13.825614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bd60df9238f4c1689138e1c28e74eca)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:13.857890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.004871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bd60df9238f4c1689138e1c28e74eca', 'request_id': 'req-e7b50306-13cd-4890-8091-1367262588a7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:14.019630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bd60df9238f4c1689138e1c28e74eca', 'request_id': 'req-e7b50306-13cd-4890-8091-1367262588a7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:14.038378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bd60df9238f4c1689138e1c28e74eca', 'request_id': 'req-e7b50306-13cd-4890-8091-1367262588a7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:14.057512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] The token cache is being invalidate because project 3bd60df9238f4c1689138e1c28e74eca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:14.057813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:14.062045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e7b50306-13cd-4890-8091-1367262588a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:14.064294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1053/2103] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:13 2022] DELETE /identity/v3/projects/3bd60df9238f4c1689138e1c28e74eca => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:14.069662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.069946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.070406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.136620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:14.150852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:14.310940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1051/2104] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:14.322365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.322822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.323115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.360104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:14.361658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:14.379386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1054/2105] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:14.383772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.399292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a4fac8-9bdb-4476-873c-00ac917e6a3f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.429694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.430871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.431266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.431694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.434423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:14.434804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.438322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.475047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1052/2106] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:14.478651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.499483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7e356-234a-4c9a-b779-71ec05e44ca9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.523352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.524220 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.524510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.524936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.525872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.528438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:14.638974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1055/2107] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:14 2022] POST /identity/v3/projects => generated 373 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:14.646695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:14.682580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d644559-f7f3-48fb-b3b4-331faae88e89 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:14.916710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:14.930479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:14.932403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:14.932862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:14.936449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:14.943704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ab6fb3-b251-461d-b2c1-bec261a0aecc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:15.021846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1053/2108] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:14 2022] POST /identity/v3/users => generated 287 bytes in 379 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:37:15.027653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:15.084936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.085281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.085591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.085849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca38dfe1-9477-4c44-a8ea-65cd4e4c8d2f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:15.100134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:15.100866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.101155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.101487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.102160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:15.104569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:15.117145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1056/2109] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:15.118128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:15.120706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:15.125075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] MFA Rules not processed for user `76b2078f1f64440e9d8faca3bcf98d52`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:15.126475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:15.148737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea149011-5202-4f12-91d5-ebdef4a2db00 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:15.163521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:15.164235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.164500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.165035 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] PATH_INFO: `/v3/projects/8feb044ea9654cb9b3f0ab0c1442b6d1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.181944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8feb044ea9654cb9b3f0ab0c1442b6d1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:15.188590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:15.335262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1054/2110] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:15.336608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:15.340195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8feb044ea9654cb9b3f0ab0c1442b6d1', 'request_id': 'req-180d8b79-872d-4e6f-a9ff-6a118856da76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:15.353633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8feb044ea9654cb9b3f0ab0c1442b6d1', 'request_id': 'req-180d8b79-872d-4e6f-a9ff-6a118856da76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:15.358711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b5f1f5-0a8a-4f58-9af1-7d54254e5196 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:15.359527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:15.367484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8feb044ea9654cb9b3f0ab0c1442b6d1', 'request_id': 'req-180d8b79-872d-4e6f-a9ff-6a118856da76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:15.375338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:15.394766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] The token cache is being invalidate because project 8feb044ea9654cb9b3f0ab0c1442b6d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:15.395249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:15.397477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-180d8b79-872d-4e6f-a9ff-6a118856da76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:15.401959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1057/2111] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:15 2022] DELETE /identity/v3/projects/8feb044ea9654cb9b3f0ab0c1442b6d1 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:15.407191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc11fc-1360-4fdc-bc04-995d15fa049e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.407957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc11fc-1360-4fdc-bc04-995d15fa049e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.409163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc11fc-1360-4fdc-bc04-995d15fa049e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.446495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-19cc11fc-1360-4fdc-bc04-995d15fa049e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:15.447777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-19cc11fc-1360-4fdc-bc04-995d15fa049e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:15.541264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:15.544318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.544496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.544747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.545660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:15.555860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:15.622540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1055/2112] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:15.628400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:15.636125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:15.667752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1058/2113] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:15.677094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.677383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.677680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.683041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c44dce-4bc5-4307-a8fe-0760559c816b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:15.701255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:15.701858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.702146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.702442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] PATH_INFO: `/v3/projects/9f23c6087a3041cc803fbb287a731494/users/8ce55c86c7b245ed9100b54d78bdc789/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.730303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:15.735517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:15.737996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f23c6087a3041cc803fbb287a731494, user_id=8ce55c86c7b245ed9100b54d78bdc789, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:15.743091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bbb60c-3295-48ba-9c14-73453965ac9d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:15.748431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1059/2114] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:15.754399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:15.780397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ade46-70e5-45f7-a581-afa0b1f34f54 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:15.783726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1056/2115] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:15 2022] PUT /identity/v3/projects/9f23c6087a3041cc803fbb287a731494/users/8ce55c86c7b245ed9100b54d78bdc789/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:15.802065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:15.803018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:15.806326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:15.928311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] MFA Rules not processed for user `8ce55c86c7b245ed9100b54d78bdc789`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:15.932127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:16.022810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.023465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.023800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.024077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.024803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:16.025255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.026834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.059678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1060/2116] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:37:16.060665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.092935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1404e2-8918-4038-93a9-869ca141ce5f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.103673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1057/2117] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:16.117132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.117659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.117976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.118236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.119106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.121442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.138252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.140833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:16.168564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1061/2118] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:16 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:16.169914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.298623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36a3ed-5bd8-4f14-b88c-4f7c11eea2c1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.302458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:16.319029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6b9808-07a5-4eae-b7b6-456dbe5d409e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.327566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.328443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.328733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.329007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.329725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.334381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.353211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:16.387129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1062/2119] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:16 2022] POST /identity/v3/users => generated 289 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:16.388660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.402526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12830f7a-df26-462b-94b5-3b9126553f79 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.412530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.414196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.415700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.416203 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.416871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.423865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.429235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.429875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.430268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.430678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.451719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1063/2120] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:16.456857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.489766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edd0214-4b47-4ded-9a70-74ea6e24857b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.508052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.508742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.509407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.509407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.515192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.537937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.544782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.547422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8bd175-bb06-48f5-873f-9f1d4f463406 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.555433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1064/2121] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:16.560442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:16.569993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1058/2122] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:16 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:16.574355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c9e500-4c38-4dd7-ac11-45afb7cdc451 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:16.581251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:16.582098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.582445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.582772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] PATH_INFO: `/v3/projects/86adf9e1bf5140399fef28e5b357ea67/users/35fbe07f324143e3b81cbcc4028f5774/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.613715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86adf9e1bf5140399fef28e5b357ea67, user_id=35fbe07f324143e3b81cbcc4028f5774, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:16.621024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:16.679133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1065/2123] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:16 2022] PUT /identity/v3/projects/86adf9e1bf5140399fef28e5b357ea67/users/35fbe07f324143e3b81cbcc4028f5774/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:16.698401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d0a2-51b7-4731-8546-dcb0fec63acf None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:16.699058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d0a2-51b7-4731-8546-dcb0fec63acf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:16.699542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d0a2-51b7-4731-8546-dcb0fec63acf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:16.805446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7082d0a2-51b7-4731-8546-dcb0fec63acf None None] MFA Rules not processed for user `35fbe07f324143e3b81cbcc4028f5774`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:16.806043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7082d0a2-51b7-4731-8546-dcb0fec63acf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:16.977300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1059/2124] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:16 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:16.982801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:17.003958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.004210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.004383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.124689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] MFA Rules not processed for user `78634b34107d49cdbf0126014bf8f6f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:17.132348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:17.359815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f22346-33bb-4d1c-912d-ed7fe91b3338 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:17.398902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:17.399548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.399824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.400089 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.400757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:17.422363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:17.482037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1060/2125] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 479 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:17.501573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1066/2126] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:16 2022] POST /identity/v3/projects => generated 377 bytes in 518 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:17.507546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:17.561597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaa963f-57dd-452c-9e04-4aa648e4cbf9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:17.600550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:17.764225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:17.765396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.766362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.767474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.768984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:17.776296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:17.838229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1061/2127] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:17 2022] POST /identity/v3/users => generated 289 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:17.842726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:17.875633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab875bc7-725a-4738-a3a2-1e0dde90748e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:17.896881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:17.901922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.903135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.903489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.908280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:17.911649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:17.970368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1067/2128] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:17.976366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:17.988611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7858d0-dbfe-4f53-9d4f-9d87458ad4a2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:17.994097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:17.994695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:17.994933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:17.996003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:17.996679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:17.998405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d5ec47-7efd-4af0-9628-80736a403945 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:18.024577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1062/2129] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:18.029263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:18.043970 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb7e120-c908-48d2-9aa8-791fa8981848 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:18.056385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:18.060532 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.061042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.061504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] PATH_INFO: `/v3/projects/62c3e2084b3e4d3aa118be14bc63596b/users/c9cf30a81cdf4644ab8ec64a61e008be/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.100626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62c3e2084b3e4d3aa118be14bc63596b, user_id=c9cf30a81cdf4644ab8ec64a61e008be, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:18.104929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:18.135090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1068/2130] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:18 2022] PUT /identity/v3/projects/62c3e2084b3e4d3aa118be14bc63596b/users/c9cf30a81cdf4644ab8ec64a61e008be/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:18.141284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.141661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.142009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.215390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] MFA Rules not processed for user `c9cf30a81cdf4644ab8ec64a61e008be`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:18.217402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:18.358795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1063/2131] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:18.366351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:18.479092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:18.506807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28795c6f-cfd9-4fc8-9ea1-ad9cc3f03372 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:18.514491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:18.515599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.516138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.516632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.517590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:18.521045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:18.568911 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1069/2132] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:18 2022] POST /identity/v3/projects => generated 375 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:18.574081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:18.602153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095a06b6-9ab9-4f08-9ef8-cc3837ff947f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:18.722374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:18.723345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.723971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.724459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.725912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:18.738294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:18.804773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:18.812630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1064/2133] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:18 2022] POST /identity/v3/users => generated 294 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:18.819213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8d4d20-2831-4261-ac8d-df5438dc5b30 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:18.824187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313af58-2927-4336-855f-34207173866e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:18.824962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313af58-2927-4336-855f-34207173866e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.825304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313af58-2927-4336-855f-34207173866e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.825660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1313af58-2927-4336-855f-34207173866e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.826382 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1313af58-2927-4336-855f-34207173866e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:18.828074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1313af58-2927-4336-855f-34207173866e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:18.867870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:18.880493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1070/2134] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:18 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:18.882808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e575c5-b09d-4c16-ae95-6c3cf10bf9e1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:18.929749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:18.930763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:18.931043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:18.931317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] PATH_INFO: `/v3/projects/f50bf84333074499b96ea5be85778d96/users/ad988168802c4dd48354949ad0624592/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:18.960244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:18.972456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f50bf84333074499b96ea5be85778d96, user_id=ad988168802c4dd48354949ad0624592, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:18.974377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:19.013961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1065/2135] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:18 2022] PUT /identity/v3/projects/f50bf84333074499b96ea5be85778d96/users/ad988168802c4dd48354949ad0624592/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:37:19.018414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313af58-2927-4336-855f-34207173866e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:19.036721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1313af58-2927-4336-855f-34207173866e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:19.233865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:19.234488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.234821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.235086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.235776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:19.237903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82967d4e-fa0c-4fda-9ac0-baa0cb0fcd83 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:19.248657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1071/2136] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:19.253485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:19.266653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12780434-e297-490a-b808-5abfc7114105 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:19.284258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:19.286653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.286899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.287175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] PATH_INFO: `/v3/projects/f50bf84333074499b96ea5be85778d96/users/ad988168802c4dd48354949ad0624592/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.291251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f50bf84333074499b96ea5be85778d96, user_id=ad988168802c4dd48354949ad0624592, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:19.295256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:19.316194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1066/2137] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:19 2022] PUT /identity/v3/projects/f50bf84333074499b96ea5be85778d96/users/ad988168802c4dd48354949ad0624592/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:19.327323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.327608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.328085 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.384111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] MFA Rules not processed for user `ad988168802c4dd48354949ad0624592`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:19.388453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:19.638794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1072/2138] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:19 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:37:19.650040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:19.652188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:19.774240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c94b1f-cb40-4495-a286-a54cae17eb18 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:19.822004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:19.822849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:19.825802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:19.826220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:19.914649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:19.916455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c41c4b9-d0c7-483a-9fdd-ec8a7032d8c4 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:19.931880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1067/2139] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:19 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:21.837651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:21.862958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbf390-454c-48f6-b2e2-cb57a253eba0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:21.868413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:21.869373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:21.869800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:21.870308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:21.927331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:21.959880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:21.961335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:21.985609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1073/2140] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:22.465543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:22.466057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:22.466057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:22.500303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] MFA Rules not processed for user `0c1de2efe2ed42508d7ab4a2e9d91e92`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:22.502115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:22.625996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1068/2141] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:22.885226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:22.894922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1245f8e7-cfa3-4163-9c05-60191be3d0bb service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:22.899785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:22.900518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:22.901184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:22.901592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:22.914286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:22.916243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835045c1-6f95-4bc7-b3ac-3b6e3c60e41f service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:22.928017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1074/2142] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:22 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:27.669573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:27.737267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:27.767989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13024bec-ec64-42d9-8787-0b9deb01d362 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:27.773731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:27.774597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:27.775055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:27.775333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] PATH_INFO: `/v3/users/35fbe07f324143e3b81cbcc4028f5774` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:27.776625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35fbe07f324143e3b81cbcc4028f5774)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:27.778637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:27.838387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35fbe07f324143e3b81cbcc4028f5774'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:27.847805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35fbe07f324143e3b81cbcc4028f5774'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:27.862988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35fbe07f324143e3b81cbcc4028f5774'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:27.863741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35fbe07f324143e3b81cbcc4028f5774'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:27.874024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35fbe07f324143e3b81cbcc4028f5774'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:27.887464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1069/2143] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:27 2022] DELETE /identity/v3/users/35fbe07f324143e3b81cbcc4028f5774 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:27.891920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:27.892472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:27.892472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:27.912296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:27.913910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:28.005803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1075/2144] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:37:28.026117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:28.027587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:28.112080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9facdf51-3fed-424e-950b-dc1fabab91b1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:28.115776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:28.116397 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:28.116671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:28.116934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:28.131235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:28.133832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:28.144707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1070/2145] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:28.587888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:28.589346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:28.614928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10572503-1b12-483c-a972-a0c4c2605bb9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:28.631480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:28.631954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:28.632230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:28.638588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] PATH_INFO: `/v3/projects/86adf9e1bf5140399fef28e5b357ea67` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:28.638588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] RBAC: Authorizing `identity:delete_project(project_id=86adf9e1bf5140399fef28e5b357ea67)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:28.638588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:28.714241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86adf9e1bf5140399fef28e5b357ea67', 'request_id': 'req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.726074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86adf9e1bf5140399fef28e5b357ea67', 'request_id': 'req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.733246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86adf9e1bf5140399fef28e5b357ea67', 'request_id': 'req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.745479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] The token cache is being invalidate because project 86adf9e1bf5140399fef28e5b357ea67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:28.746034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.747657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.750281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1076/2146] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:28 2022] DELETE /identity/v3/projects/86adf9e1bf5140399fef28e5b357ea67 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:28.754910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:28.764826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508eb115-ecdc-4509-b187-9b6fe44d5274 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:28.830601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:28.839233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:28.840111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:28.840602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:28.846150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] PATH_INFO: `/v3/users/c9cf30a81cdf4644ab8ec64a61e008be` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:28.848402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9cf30a81cdf4644ab8ec64a61e008be)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:28.850521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:28.916725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9cf30a81cdf4644ab8ec64a61e008be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.924695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9cf30a81cdf4644ab8ec64a61e008be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.946810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9cf30a81cdf4644ab8ec64a61e008be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.947555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9cf30a81cdf4644ab8ec64a61e008be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.957880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9cf30a81cdf4644ab8ec64a61e008be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:28.964682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1071/2147] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:28 2022] DELETE /identity/v3/users/c9cf30a81cdf4644ab8ec64a61e008be => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:29.423316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:29.426718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.446927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf4b68-4568-4bdf-aea0-1a07d4faf76b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:29.557162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:29.579735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:29.582765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:29.584400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:29.584933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] PATH_INFO: `/v3/projects/62c3e2084b3e4d3aa118be14bc63596b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:29.587408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] RBAC: Authorizing `identity:delete_project(project_id=62c3e2084b3e4d3aa118be14bc63596b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:29.589361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:29.630848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:29.631821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.646892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d28fe-1488-4970-98ff-72f8bff3c79b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:29.661974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:29.661974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:29.661974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:29.661974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] PATH_INFO: `/v3/users/8ce55c86c7b245ed9100b54d78bdc789` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:29.662422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ce55c86c7b245ed9100b54d78bdc789)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:29.662422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:29.687992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62c3e2084b3e4d3aa118be14bc63596b', 'request_id': 'req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.698753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62c3e2084b3e4d3aa118be14bc63596b', 'request_id': 'req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.709956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62c3e2084b3e4d3aa118be14bc63596b', 'request_id': 'req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.724579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ce55c86c7b245ed9100b54d78bdc789'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.729965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] The token cache is being invalidate because project 62c3e2084b3e4d3aa118be14bc63596b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:29.730451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.731420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-75be2eb7-cfbe-411a-8d44-56ebb0a7413b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.734097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ce55c86c7b245ed9100b54d78bdc789'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.734451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1077/2148] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:29 2022] DELETE /identity/v3/projects/62c3e2084b3e4d3aa118be14bc63596b => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:29.736357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412b44-f4d2-4f6f-b21a-08d15c4642f9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:29.737342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412b44-f4d2-4f6f-b21a-08d15c4642f9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:29.737628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412b44-f4d2-4f6f-b21a-08d15c4642f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:29.758603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ce55c86c7b245ed9100b54d78bdc789'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.758603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ce55c86c7b245ed9100b54d78bdc789'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.762244 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-85412b44-f4d2-4f6f-b21a-08d15c4642f9 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:29.763998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-85412b44-f4d2-4f6f-b21a-08d15c4642f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.773909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ce55c86c7b245ed9100b54d78bdc789'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:29.782494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1072/2149] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:29 2022] DELETE /identity/v3/users/8ce55c86c7b245ed9100b54d78bdc789 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:29.785018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:29.788338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.859737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1078/2150] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:37:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:29.861293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:29.861700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:29.862102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:29.888311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:29.910219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df4c32-30fd-4536-ba02-fd841e234561 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:29.916060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:29.918016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.921304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:29.923146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:29.923721 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:29.924265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] PATH_INFO: `/v3/users/ad988168802c4dd48354949ad0624592` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:29.927286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad988168802c4dd48354949ad0624592)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:29.929982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:29.930182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1079/2151] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:29.937440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:29.945479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:29.997794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad988168802c4dd48354949ad0624592'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:30.007189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad988168802c4dd48354949ad0624592'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:30.031229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad988168802c4dd48354949ad0624592'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:30.032236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad988168802c4dd48354949ad0624592'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:30.037042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50390312-e713-459c-a121-1c1f26b6c6ca None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:30.043107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:30.043794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:30.044037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:30.044280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:30.046133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:30.046607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad988168802c4dd48354949ad0624592'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:30.057555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1073/2152] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:29 2022] DELETE /identity/v3/users/ad988168802c4dd48354949ad0624592 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:30.064811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:30.067175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:30.093633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:30.181080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b887f9b8-b7dc-4251-a557-e3f8a1b942a3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:30.191370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:30.192358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:30.193509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:30.193846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:30.199675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:30.210270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:30.212076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:30.233008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1080/2153] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:37:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:37:30.408738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:30.411563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:30.433805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1074/2154] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:30.987741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:30.989944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:31.014615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766acdca-87ac-437a-9f8e-039da1af61af service swift] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:31.036057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:31.036735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.037023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.037300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] PATH_INFO: `/v3/projects/f50bf84333074499b96ea5be85778d96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.040896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f50bf84333074499b96ea5be85778d96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:31.047325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:31.215288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f50bf84333074499b96ea5be85778d96', 'request_id': 'req-6c2a088d-e6a8-4976-aa33-2df5efc9a240'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.229858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f50bf84333074499b96ea5be85778d96', 'request_id': 'req-6c2a088d-e6a8-4976-aa33-2df5efc9a240'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.253262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f50bf84333074499b96ea5be85778d96', 'request_id': 'req-6c2a088d-e6a8-4976-aa33-2df5efc9a240'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.326437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] The token cache is being invalidate because project f50bf84333074499b96ea5be85778d96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:31.327575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.330052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6c2a088d-e6a8-4976-aa33-2df5efc9a240 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.350561 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:31.353675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:31.381553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed85043-85d4-4433-9e46-f4832744061c service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:31.393481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1081/2155] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:30 2022] DELETE /identity/v3/projects/f50bf84333074499b96ea5be85778d96 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:31.401901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fe09-5f35-40f9-8be8-607678b7f2a1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.402261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fe09-5f35-40f9-8be8-607678b7f2a1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.402443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f1fe09-5f35-40f9-8be8-607678b7f2a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.429120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:31.452036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-16f1fe09-5f35-40f9-8be8-607678b7f2a1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:31.453425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-16f1fe09-5f35-40f9-8be8-607678b7f2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:31.546755 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:31.547962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.550905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.551842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] PATH_INFO: `/v3/projects/9f23c6087a3041cc803fbb287a731494` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.554614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f23c6087a3041cc803fbb287a731494)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:31.560269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:31.591641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1082/2156] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:31.599505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.599956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.600243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.639835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:31.641250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:31.657232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1083/2157] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:31.663214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:31.681051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7126f31-8be7-451e-9cb0-ed7b4d2e060c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:31.709032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:31.709660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.709933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.710205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.710895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:31.711196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:31.712934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:31.742572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f23c6087a3041cc803fbb287a731494', 'request_id': 'req-7104c428-5980-4479-b4da-7ec6ea17d704'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.756711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1084/2158] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:31.757927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:31.768076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f23c6087a3041cc803fbb287a731494', 'request_id': 'req-7104c428-5980-4479-b4da-7ec6ea17d704'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.777250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22cb54b-0b57-48da-b1b7-9d3789a50ec0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:31.790960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f23c6087a3041cc803fbb287a731494', 'request_id': 'req-7104c428-5980-4479-b4da-7ec6ea17d704'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.803318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:31.841146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] The token cache is being invalidate because project 9f23c6087a3041cc803fbb287a731494 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:31.841563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.842775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7104c428-5980-4479-b4da-7ec6ea17d704 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:31.856849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1075/2159] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:31 2022] DELETE /identity/v3/projects/9f23c6087a3041cc803fbb287a731494 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:31.858377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e8441-b8e1-4f47-9805-82b2ba4b5f63 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.858693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e8441-b8e1-4f47-9805-82b2ba4b5f63 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.858972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e8441-b8e1-4f47-9805-82b2ba4b5f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.859683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:31.860299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:31.860579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:31.860848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:31.862497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:31.873417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:31.943731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8d1e8441-b8e1-4f47-9805-82b2ba4b5f63 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:31.944983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8d1e8441-b8e1-4f47-9805-82b2ba4b5f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:31.949762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1085/2160] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:31 2022] POST /identity/v3/projects => generated 373 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:31.960377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:31.966121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:32.107716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1076/2161] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:32.113371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.113761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.114042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.139320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:32.157662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:32.176460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d6236-108f-4f36-981c-7e55fea342e6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.187589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1077/2162] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:32.197827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.198460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.198629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.206831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.207435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.208154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.215866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.228261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d67f40e-80f4-486b-bd01-d8d8124e505c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.245525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.248014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.248333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.248713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.249768 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:37:32.253093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.260588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.271049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1086/2163] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:31 2022] POST /identity/v3/users => generated 287 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:32.272774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.291700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f2666-6351-4afb-b9dc-7455546f7aba admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.297808 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.298489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.299435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.299764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.300629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.302309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.309038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1078/2164] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:37:32.310344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.336266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f911f-12ff-446f-bda2-bb8fbd4394d0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.337251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1087/2165] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:32.341882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.344057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.344678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.344956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.345283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.346460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.349816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.368706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f18f6e-2d4b-4a20-86d6-c0216d72a1c5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.375857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.376912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.377586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.378624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.379486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.381896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.400731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1088/2166] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:32.402456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.424436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d44d9a-6efc-4ff5-b641-3fc9d6277c15 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.431873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1079/2167] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:32 2022] POST /identity/v3/projects => generated 399 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:32.437202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.463607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:32.553392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.554087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.554363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.554819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] PATH_INFO: `/v3/projects/ed407864f9cf431ebd8018e1a6862294/users/d1520d0cbea447a3a44a992ca10ca587/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.585358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed407864f9cf431ebd8018e1a6862294, user_id=d1520d0cbea447a3a44a992ca10ca587, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.586821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef50c36-8261-4d7e-93ea-66bc8db3dd48 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.588356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134042e7-1036-4a3c-af4b-3ae53c88c034 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.601288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:32.612179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.616967 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1089/2168] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:32 2022] PUT /identity/v3/projects/ed407864f9cf431ebd8018e1a6862294/users/d1520d0cbea447a3a44a992ca10ca587/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:32.617777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.618132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.618346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.619171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.621664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.624185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.624825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.625658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.655408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:32.657751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:32.666785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1080/2169] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:32 2022] POST /identity/v3/users => generated 300 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:32.668173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.682595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25c8dc0-1f30-499b-b17b-32b854f0375d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.764622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.786806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1090/2170] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:32.787894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.788384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.788847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.789814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.792379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.797654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.800077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:32.806970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1081/2171] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:32.811333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.827802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3317c80f-3156-4ea4-ac2c-46267d104eb7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.837832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.839541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.840510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.841197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.842173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.844786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.862381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1082/2172] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:32.867126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:32.887692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b81c1-8ba6-4f85-b55d-7837ae137213 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:32.893892 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:32.895337 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.895853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.896513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] PATH_INFO: `/v3/projects/d994cc98db1b4cfd8cfafc87cb80bdfb/users/dd5c41a5ccfb4f98b99573dbbd9c4766/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.922642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d994cc98db1b4cfd8cfafc87cb80bdfb, user_id=dd5c41a5ccfb4f98b99573dbbd9c4766, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:32.939795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa09a7e3-583d-4578-bff2-d3ee77df8ff9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:32.973410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1083/2173] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:32 2022] PUT /identity/v3/projects/d994cc98db1b4cfd8cfafc87cb80bdfb/users/dd5c41a5ccfb4f98b99573dbbd9c4766/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:32.983992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:32.985933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:32.986896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:32.987671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97199ba9-e368-41a5-995b-16ba5ee99322 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:33.058310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] MFA Rules not processed for user `dd5c41a5ccfb4f98b99573dbbd9c4766`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:33.061078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:33.105405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:33.106029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:33.106411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:33.106721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:33.160702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:33.196392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1084/2174] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:32 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:37:33.214994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:33.222270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:33.279502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:33.284514 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c418077a-1049-4cc2-b6ce-fe5702b882d2 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:33.302313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1091/2175] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:37:33.354848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf46e47-d9b9-4f89-b9b2-0991e9f77c36 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:33.370087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:33.373211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:33.373649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:33.374844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:33.395346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:33.400145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:33.440197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1085/2176] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:33 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:35.543609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b502753b-dd7f-47d6-ac78-a79c61b25d99 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:35.544174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b502753b-dd7f-47d6-ac78-a79c61b25d99 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:35.544174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b502753b-dd7f-47d6-ac78-a79c61b25d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:35.579795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:35.601555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b502753b-dd7f-47d6-ac78-a79c61b25d99 None None] MFA Rules not processed for user `ab4b3c2b3f4b4978b0eb472f9c2736b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:35.604162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b502753b-dd7f-47d6-ac78-a79c61b25d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:35.844322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1092/2177] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 04 08:37:41.920329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:41.939905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd33f5df-aabb-4c64-bb33-49397c370454 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:41.946218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:41.947143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:41.947899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:41.948135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] PATH_INFO: `/v3/users/dd5c41a5ccfb4f98b99573dbbd9c4766` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:41.950261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd5c41a5ccfb4f98b99573dbbd9c4766)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:41.952977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:42.085241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd5c41a5ccfb4f98b99573dbbd9c4766'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:42.098818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd5c41a5ccfb4f98b99573dbbd9c4766'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:42.120862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd5c41a5ccfb4f98b99573dbbd9c4766'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:42.121936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd5c41a5ccfb4f98b99573dbbd9c4766'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:42.156934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd5c41a5ccfb4f98b99573dbbd9c4766'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:42.167451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1086/2178] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:37:41 2022] DELETE /identity/v3/users/dd5c41a5ccfb4f98b99573dbbd9c4766 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:42.172566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:42.173109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:42.173109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:42.230230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:42.234793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:42.415320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1093/2179] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:42.426130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:42.428289 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:42.538424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575ecc9-3715-4ce7-a9e4-52c23417d106 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:42.555895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:42.557162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:42.557580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:42.557862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:42.578063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:42.581009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d555042a-8aa7-4872-b0d7-62ab2c2e955a service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:42.598787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1087/2180] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:43.270090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:43.272306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:43.312068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25abda4-d452-492e-955a-f7231cda0500 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:43.344137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:43.346968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:43.347662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:43.351588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] PATH_INFO: `/v3/projects/d994cc98db1b4cfd8cfafc87cb80bdfb` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:43.358381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d994cc98db1b4cfd8cfafc87cb80bdfb)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:43.367251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:43.545550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd994cc98db1b4cfd8cfafc87cb80bdfb', 'request_id': 'req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:43.559494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd994cc98db1b4cfd8cfafc87cb80bdfb', 'request_id': 'req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:43.576464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd994cc98db1b4cfd8cfafc87cb80bdfb', 'request_id': 'req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:43.606098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] The token cache is being invalidate because project d994cc98db1b4cfd8cfafc87cb80bdfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:37:43.606443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:43.607137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:37:43.608892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1094/2181] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:37:43 2022] DELETE /identity/v3/projects/d994cc98db1b4cfd8cfafc87cb80bdfb => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:45.698836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:45.699443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:45.699443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:45.707956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:45.749138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] MFA Rules not processed for user `d1520d0cbea447a3a44a992ca10ca587`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:45.751156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:45.961992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:45.963431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1088/2182] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:45.965462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:46.072168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:46.107473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbcde4-4312-4c9f-b1f1-1faaa58ca229 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.112411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.113330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.113777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.114204 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.115132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.117276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.165872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1095/2183] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:37:45 2022] POST /identity/v3/projects => generated 373 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:46.171795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:46.203486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1d7ecd-4d90-4f2b-a40b-91caa04412c4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.359101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.359740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.360065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.360284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.361494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.365502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.422998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1089/2184] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:37:46 2022] POST /identity/v3/users => generated 293 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:37:46.428247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:46.464414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7685506c-b352-427c-9e51-a086706ddcf5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.478639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.478639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.478639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.478639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.479129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.488990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.518638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1096/2185] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:37:46.522543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:46.540080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e0da0-f9fa-47f6-8c10-aee72e079bd2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.550642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.551722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.552097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.552488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] PATH_INFO: `/v3/projects/f2ccb8caf53844bebcbe26010b009e00/users/a684dbd081a044e693e1197f757b6658/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.578486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ccb8caf53844bebcbe26010b009e00, user_id=a684dbd081a044e693e1197f757b6658, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.582254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.613048 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1090/2186] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:46 2022] PUT /identity/v3/projects/f2ccb8caf53844bebcbe26010b009e00/users/a684dbd081a044e693e1197f757b6658/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:46.616349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:46.641360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd8f7a-5dbd-450a-a10d-e46e0a489739 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.752458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.753416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.754135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.754496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.755291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.757866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f25507f-5af6-45c8-aa5a-4d9616b55378 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.778600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1097/2187] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:37:46.782668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:46.801305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf2d3b-eaae-40f5-857c-2de62d542101 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:46.810151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:46.811271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:46.812129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:46.812913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] PATH_INFO: `/v3/projects/f2ccb8caf53844bebcbe26010b009e00/users/a684dbd081a044e693e1197f757b6658/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:46.817843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2ccb8caf53844bebcbe26010b009e00, user_id=a684dbd081a044e693e1197f757b6658, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:46.821233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09798462-df21-426f-90fc-46dd11fc829d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:46.853422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1091/2188] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:37:46 2022] PUT /identity/v3/projects/f2ccb8caf53844bebcbe26010b009e00/users/a684dbd081a044e693e1197f757b6658/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:37:50.182988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:50.182988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:50.182988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:50.271870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] MFA Rules not processed for user `76b2078f1f64440e9d8faca3bcf98d52`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:50.273635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:50.534582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1098/2189] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:56.833384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=74,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:58.552140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46471377-f5b2-4a23-b94a-baa619d23c09 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:58.552828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46471377-f5b2-4a23-b94a-baa619d23c09 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:58.552828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46471377-f5b2-4a23-b94a-baa619d23c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:58.607473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-46471377-f5b2-4a23-b94a-baa619d23c09 None None] MFA Rules not processed for user `a684dbd081a044e693e1197f757b6658`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:58.609008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-46471377-f5b2-4a23-b94a-baa619d23c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:58.816989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1092/2190] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:58 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:37:58.827103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:37:58.830465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:58.977052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:37:59.049203 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b7cfdb-951d-49cd-8873-751b04888e18 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:37:59.062201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:37:59.064520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:59.065015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:59.065555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:59.206424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:37:59.208546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274c7320-1307-4ab3-adf5-1909de51b59b service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:37:59.221766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1099/2191] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:37:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:37:59.669324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31c1bd-b40c-4dce-a0bc-772d2ea3cd5d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:37:59.669891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31c1bd-b40c-4dce-a0bc-772d2ea3cd5d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:37:59.670577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd31c1bd-b40c-4dce-a0bc-772d2ea3cd5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:37:59.703345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fd31c1bd-b40c-4dce-a0bc-772d2ea3cd5d None None] MFA Rules not processed for user `26abb94392364bdc82f6bb508a611d44`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:37:59.705690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fd31c1bd-b40c-4dce-a0bc-772d2ea3cd5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:37:59.815965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1093/2192] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:37:59 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:38:01.771820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf833c-9649-4426-a203-b7d4c0439fdb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:01.772300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf833c-9649-4426-a203-b7d4c0439fdb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:01.772449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf833c-9649-4426-a203-b7d4c0439fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:01.794784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cebf833c-9649-4426-a203-b7d4c0439fdb None None] MFA Rules not processed for user `d1520d0cbea447a3a44a992ca10ca587`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:01.796636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cebf833c-9649-4426-a203-b7d4c0439fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:01.806594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1100/2193] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:38:01 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:05.872259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96979-2704-474a-9272-66610b6754ec None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:05.873455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96979-2704-474a-9272-66610b6754ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:05.874429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f96979-2704-474a-9272-66610b6754ec None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:05.878358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1094/2194] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:38:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:38:05.887918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a8a2-e134-4aae-bdba-8add07ecd401 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:05.888933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a8a2-e134-4aae-bdba-8add07ecd401 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:05.889437 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a8a2-e134-4aae-bdba-8add07ecd401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:05.914821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9412a8a2-e134-4aae-bdba-8add07ecd401 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:05.916495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9412a8a2-e134-4aae-bdba-8add07ecd401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:05.931405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1101/2195] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:38:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:05.948197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f96979-2704-474a-9272-66610b6754ec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:05.950099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a3f96979-2704-474a-9272-66610b6754ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:06.051050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f96979-2704-474a-9272-66610b6754ec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:06.054986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:06.055704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.056074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.056420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.067964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:06.069467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e0fcf-f8e9-4e90-b5b8-32002e4e5518 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:06.085414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1095/2196] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:38:06.184184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.184488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.184995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.186512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1102/2197] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:38:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:38:06.192254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee32925-5d4e-4016-86a5-6c55b2699d0a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.192922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee32925-5d4e-4016-86a5-6c55b2699d0a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.193418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee32925-5d4e-4016-86a5-6c55b2699d0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.215043 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-aee32925-5d4e-4016-86a5-6c55b2699d0a None None] MFA Rules not processed for user `3e66140a80f641f4afe94ed0452000f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:06.216264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aee32925-5d4e-4016-86a5-6c55b2699d0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:06.229506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:06.278854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1096/2198] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:38:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:06.289048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:06.290846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:06.307823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dcd14c-2008-4ffe-b358-10c6dd6dc505 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:06.311621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:06.312234 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.312510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.313017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.321888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:06.323692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3aca98-b0cd-4613-9714-68a078afe9a2 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:06.333807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1103/2199] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:38:06.440339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526c8ef-018b-42e3-b8c3-07ae11849f24 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.440748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526c8ef-018b-42e3-b8c3-07ae11849f24 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.440945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3526c8ef-018b-42e3-b8c3-07ae11849f24 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.443028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1097/2200] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:38:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:38:06.449471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.449779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.450174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.479266 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:06.480737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:06.491031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1104/2201] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:38:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:38:06.503586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526c8ef-018b-42e3-b8c3-07ae11849f24 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:06.520602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3526c8ef-018b-42e3-b8c3-07ae11849f24 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:06.526086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:06.527107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:06.527392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:06.527678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:06.540871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:06.542379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feea5cf-aee5-450f-8472-e43507e86806 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:06.559800 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1098/2202] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:38:12.012221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:12.021227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecd20b4-0b1d-464a-be64-050f2a055230 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:12.025338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:12.025937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:12.026590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:12.026867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:12.028655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:12.110207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:12.111895 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:12.125539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1105/2203] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:12 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:38:14.951955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:14.953828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:14.954536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:14.976771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:14.977993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:15.079158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1099/2204] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:38:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:15.093368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:15.096958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:15.224197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:15.260973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4df9c47-82a3-44fd-84f0-ba4bc720e21c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:15.268457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:15.269162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:15.269443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:15.269717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:15.278372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:15.279996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724079c8-340f-4c9c-b145-a9494e8a1bea service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:15.289792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1106/2205] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:38:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:38:15.517493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:15.519178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:15.756672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a7e559-e5d6-4823-97ed-518cb38a8232 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:15.776581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:15.777212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:15.777528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:15.777813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:15.783684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:15.799719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:15.801595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:15.836594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1100/2206] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:38:25.286178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:25.812424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=80 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:37.236104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffdcc3-2887-4b02-af3b-a858cbe1cf9f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:37.236766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffdcc3-2887-4b02-af3b-a858cbe1cf9f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:37.236766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ffdcc3-2887-4b02-af3b-a858cbe1cf9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:37.259353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-56ffdcc3-2887-4b02-af3b-a858cbe1cf9f None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:37.260584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-56ffdcc3-2887-4b02-af3b-a858cbe1cf9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:37.269093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1107/2207] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:38:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:37.292081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:37.304189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612f0ed5-c1f0-48a4-9cae-6d1f023e2c90 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:37.309296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:37.309913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:37.310184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:37.310449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:37.325267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:37.327146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:37.339277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1101/2208] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:38:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:38:42.768941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2bc1c-772a-4588-8e7c-f201ce3427cb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:42.770736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2bc1c-772a-4588-8e7c-f201ce3427cb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:42.771408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e2bc1c-772a-4588-8e7c-f201ce3427cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:42.822301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b4e2bc1c-772a-4588-8e7c-f201ce3427cb None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:42.825416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b4e2bc1c-772a-4588-8e7c-f201ce3427cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:42.838582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1108/2209] 10.222.0.16 () {58 vars in 967 bytes} [Tue Oct 4 08:38:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:42.855177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:42.868941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a787a16f-5f24-4147-9af0-5ca21b1065b9 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:42.879423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:42.880244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:42.880608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:42.880843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:42.894114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:42.895774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:42.904918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1102/2210] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:38:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 04 08:38:52.806297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:52.901142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=12 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:38:56.067735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ef4ce-3027-433b-b4ee-6377abffe4e7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:56.068345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ef4ce-3027-433b-b4ee-6377abffe4e7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:56.068345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ef4ce-3027-433b-b4ee-6377abffe4e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:56.095236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-538ef4ce-3027-433b-b4ee-6377abffe4e7 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:56.096460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-538ef4ce-3027-433b-b4ee-6377abffe4e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:56.103954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1109/2211] 10.222.0.16 () {58 vars in 967 bytes} [Tue Oct 4 08:38:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:38:56.118022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:56.130731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9fa6ad-7765-4777-a252-c513dca8f43d service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:56.138486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:56.139130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:56.139412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:56.139675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:56.148980 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:56.150631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a693452-a9b8-4f6a-91e8-709db83b20fe service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:56.161370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1103/2212] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:38:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:38:58.701847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:58.702338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:58.702420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:58.704579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1110/2213] 10.222.0.16 () {54 vars in 921 bytes} [Tue Oct 4 08:38:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:38:59.048867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:59.052749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:59.053659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:59.115522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:59.123739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:59.129778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:38:59.131396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:38:59.152204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1104/2214] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Oct 4 08:38:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:38:59.163645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:59.178415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9406cd5-7c24-4415-9719-2105f46d9430 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:59.184731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:59.185972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:59.186695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:59.188240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:59.201883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:59.205541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:59.232193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1105/2215] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:38:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:38:59.323462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1d37a5-9201-4ad1-a55d-c56f0c8642f5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:59.329971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:59.330977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:59.331303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:59.331626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] PATH_INFO: `/v3/users/76b2078f1f64440e9d8faca3bcf98d52` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:59.333060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] RBAC: Authorizing `identity:delete_user(user_id=76b2078f1f64440e9d8faca3bcf98d52)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:59.335055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:59.420332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76b2078f1f64440e9d8faca3bcf98d52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.429580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76b2078f1f64440e9d8faca3bcf98d52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.445310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76b2078f1f64440e9d8faca3bcf98d52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.446269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76b2078f1f64440e9d8faca3bcf98d52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.446991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0a1fac-a281-4a8e-8f4b-bc7251178875 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76b2078f1f64440e9d8faca3bcf98d52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.465101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1111/2216] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:38:59 2022] DELETE /identity/v3/users/76b2078f1f64440e9d8faca3bcf98d52 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:38:59.788495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:38:59.810463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc5697b-f1b5-4fec-b40c-0f385a7aec11 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:38:59.907451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:38:59.908122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:38:59.908415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:38:59.908896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] PATH_INFO: `/v3/projects/f758e7615232486a93a915760633b702` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:38:59.910176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f758e7615232486a93a915760633b702)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:38:59.912664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:38:59.988955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f758e7615232486a93a915760633b702', 'request_id': 'req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:38:59.996581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f758e7615232486a93a915760633b702', 'request_id': 'req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.008320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f758e7615232486a93a915760633b702', 'request_id': 'req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.021435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] The token cache is being invalidate because project f758e7615232486a93a915760633b702 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:39:00.021781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.022411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-eb6240ed-3653-4ef2-8e95-f35bbf9f486b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.026625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1106/2217] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:38:59 2022] DELETE /identity/v3/projects/f758e7615232486a93a915760633b702 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:39:00.029827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.030410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.030714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.053615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:00.054879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:00.122756 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1112/2218] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:00.131811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.132499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.132499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.153714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:00.155117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:00.164075 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1107/2219] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:00.168074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.180554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ef548-3587-4e8f-9147-26389067d21a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.186216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.187188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.187711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.187711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.188503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:39:00.188916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.191524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.199920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:00.211554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1113/2220] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:39:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:39:00.213208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.226517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ee49c3-2c83-4010-a5c7-6b4167661690 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.231349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.231958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.232339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.232545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.233378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.235532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.268336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1108/2221] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/projects => generated 373 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:39:00.272771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.290320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ee6c5f-ae15-4865-8ea1-abb4eb00bff3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.355270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.356130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.356591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.356860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.357937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.359759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.388230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1114/2222] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/users => generated 287 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:39:00.392852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.404010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91cc1a-2101-457b-a284-07414fb32219 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.410228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.410993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.411334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.411661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.412379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.414198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.426873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1109/2223] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:39:00.431017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.442738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3b8280-2dba-4cc8-b2b1-6bf309516fe1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.447407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.448011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.448282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.448543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.449180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.450800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1efb101-5a17-4335-90da-1580b90b48cd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.463610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1115/2224] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:39:00.467586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.478474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a51424-a2a5-46e7-8a2f-a7429ef14bc7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.483270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.484148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.484440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.484620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] PATH_INFO: `/v3/projects/f5ac5d01641443f982c5080310f0e7cd/users/84f8b3d30469496e9a22a540a1869ddd/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.501549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5ac5d01641443f982c5080310f0e7cd, user_id=84f8b3d30469496e9a22a540a1869ddd, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.504278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.506528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:00.521193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1110/2225] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:39:00 2022] PUT /identity/v3/projects/f5ac5d01641443f982c5080310f0e7cd/users/84f8b3d30469496e9a22a540a1869ddd/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:00.531540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.531830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.532095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.561298 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] MFA Rules not processed for user `84f8b3d30469496e9a22a540a1869ddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:00.563103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:00.634274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1116/2226] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:00.645760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.647834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:00.717654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3816a928-ddf1-4627-bb34-a1dc008698c6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.721999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.722830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.723227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.723596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.733222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.735463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85651c21-19f8-40fb-aa5f-4a2370292ccb service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.745779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1111/2227] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:39:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:39:00.775755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:00.863614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232caf75-92fa-4952-8f45-411a84cd8482 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:00.869434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:00.870067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.870344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.870651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] PATH_INFO: `/v3/users/84f8b3d30469496e9a22a540a1869ddd` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:00.871835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=84f8b3d30469496e9a22a540a1869ddd)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:00.874027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:00.888512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:00.936017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84f8b3d30469496e9a22a540a1869ddd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.943623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84f8b3d30469496e9a22a540a1869ddd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.958051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84f8b3d30469496e9a22a540a1869ddd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.958819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84f8b3d30469496e9a22a540a1869ddd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.959438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84f8b3d30469496e9a22a540a1869ddd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:00.977345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1117/2228] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:39:00 2022] DELETE /identity/v3/users/84f8b3d30469496e9a22a540a1869ddd => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:00.981165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:00.981459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:00.981736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.005235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:01.006695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:01.076801 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1112/2229] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:01.087135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.088586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:01.153971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f00151-1061-47c7-aa3b-d8a0944d5cfb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.158456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.159108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.159385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.159654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.170165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.172156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b91a6e0-5166-42f8-a794-61773a9e979f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.183633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1118/2230] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:39:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:39:01.491222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.492798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:01.506000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7435e6-980c-406d-ba14-7767df5b054b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.510788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.511431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.511700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.512173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] PATH_INFO: `/v3/projects/f5ac5d01641443f982c5080310f0e7cd` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.513460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5ac5d01641443f982c5080310f0e7cd)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.515435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.564712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5ac5d01641443f982c5080310f0e7cd', 'request_id': 'req-a540d072-45a3-4dac-8731-a56df892c385'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:01.570963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5ac5d01641443f982c5080310f0e7cd', 'request_id': 'req-a540d072-45a3-4dac-8731-a56df892c385'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:01.579494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5ac5d01641443f982c5080310f0e7cd', 'request_id': 'req-a540d072-45a3-4dac-8731-a56df892c385'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:01.590236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] The token cache is being invalidate because project f5ac5d01641443f982c5080310f0e7cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:39:01.590524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:01.591205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a540d072-45a3-4dac-8731-a56df892c385 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:01.592940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1113/2231] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:39:01 2022] DELETE /identity/v3/projects/f5ac5d01641443f982c5080310f0e7cd => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:01.597963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.598242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.598509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.618703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:01.619912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:01.681507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1119/2232] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:01.691012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.691344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.691658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.713476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:01.714733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:01.723408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1114/2233] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:01.727747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.738505 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bedb702-6844-4f2d-870a-e287ab6d457a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.743123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.743726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.744000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.744285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.744922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:39:01.745218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.746830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.759964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1120/2234] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:39:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:39:01.765616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.776225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99ea2d9-ca3a-446d-bbfc-6d5a789de33b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.781182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.781975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.782246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.782512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.783175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.785142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.797354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:01.815842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1115/2235] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:39:01 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:39:01.820049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.830831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80687146-c91e-47ca-8b7d-6c926b587a6a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.843357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:01.896724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.897403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.897687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.897957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.898659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.900613 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.927457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1121/2236] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:39:01 2022] POST /identity/v3/users => generated 291 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:39:01.933300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.950060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc2c4-35a2-4371-b646-c263d320bcd8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.956290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.956936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.957213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.957484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.958149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.959858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:01.971795 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1116/2237] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:39:01.976309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:01.988327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cebac16-b9fb-4ae3-9e9d-60cb96a0f4f1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:01.993344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:01.993960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:01.994233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:01.994500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:01.995182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:01.996855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36505d4a-1322-4e44-bba1-ef02c2477b00 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:02.009885 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1122/2238] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:39:02.014269 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:02.024712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ca73d-fc98-4efc-ba6a-60bf1b935fdd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:02.030975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:02.031707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:02.032055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:02.037844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] PATH_INFO: `/v3/projects/f0b762e9e807416b925d8d34bb16fbf0/users/9c42d5aec138420a90b87226dbe6dc20/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:02.054896 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0b762e9e807416b925d8d34bb16fbf0, user_id=9c42d5aec138420a90b87226dbe6dc20, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:02.057558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:02.077688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1117/2239] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:39:02 2022] PUT /identity/v3/projects/f0b762e9e807416b925d8d34bb16fbf0/users/9c42d5aec138420a90b87226dbe6dc20/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:02.084938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff0a0-3a34-49a6-a0d9-d06757aa0f89 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:02.085691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff0a0-3a34-49a6-a0d9-d06757aa0f89 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:02.086138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff0a0-3a34-49a6-a0d9-d06757aa0f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:02.113181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d33ff0a0-3a34-49a6-a0d9-d06757aa0f89 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:02.114464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d33ff0a0-3a34-49a6-a0d9-d06757aa0f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:02.178524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1123/2240] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:39:02.188155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:02.189941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:02.252886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4346c41-6f86-4c1e-83ac-991f0a74bccb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:02.256616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:02.257229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:02.257505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:02.257783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:02.268156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:02.269621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:02.280047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1118/2241] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:39:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:39:03.061252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d69f3-78bd-4e54-892c-fc83d0896e92 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:03.061539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d69f3-78bd-4e54-892c-fc83d0896e92 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:03.062395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2d69f3-78bd-4e54-892c-fc83d0896e92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:03.091705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9d2d69f3-78bd-4e54-892c-fc83d0896e92 None None] MFA Rules not processed for user `9c42d5aec138420a90b87226dbe6dc20`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:03.092933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9d2d69f3-78bd-4e54-892c-fc83d0896e92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:03.168747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1124/2242] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:03 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:03.180190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:03.181686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:03.243333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f24c7-727a-4d0f-8391-af628d652468 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:03.246961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:03.247568 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:03.247915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:03.248149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:03.258371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:03.259888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898787d8-ea21-4506-8a92-e93418c02fe5 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:03.269474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1119/2243] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:39:03 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:39:04.684381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:04.684919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:04.684919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:04.706603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] MFA Rules not processed for user `9c42d5aec138420a90b87226dbe6dc20`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:04.707754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:04.716573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1125/2244] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:10.176541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:10.177155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:10.177550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:10.210482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] MFA Rules not processed for user `0c1de2efe2ed42508d7ab4a2e9d91e92`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:10.212602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:10.279227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1120/2245] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:14.700359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:20.272137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:29.248421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:29.250279 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:29.265376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebff419e-6ffa-4d4f-8bd0-6cada3300eda None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:29.270055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:29.270707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:29.270988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:29.271259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] PATH_INFO: `/v3/users/d1520d0cbea447a3a44a992ca10ca587` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:29.272419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1520d0cbea447a3a44a992ca10ca587)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:29.274388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:29.331968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1520d0cbea447a3a44a992ca10ca587'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.339535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1520d0cbea447a3a44a992ca10ca587'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.356705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1520d0cbea447a3a44a992ca10ca587'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.357452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1520d0cbea447a3a44a992ca10ca587'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.358784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1520d0cbea447a3a44a992ca10ca587'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.373228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1126/2246] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:39:29 2022] DELETE /identity/v3/users/d1520d0cbea447a3a44a992ca10ca587 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:29.562537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:29.574410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ede04d-e24a-495d-9743-bb1c6e0ee7ad None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:29.642750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:29.643440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:29.643878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:29.645239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] PATH_INFO: `/v3/projects/ed407864f9cf431ebd8018e1a6862294` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:29.646431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed407864f9cf431ebd8018e1a6862294)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:29.648425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:29.698137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed407864f9cf431ebd8018e1a6862294', 'request_id': 'req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.704909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed407864f9cf431ebd8018e1a6862294', 'request_id': 'req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.713668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed407864f9cf431ebd8018e1a6862294', 'request_id': 'req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.725645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] The token cache is being invalidate because project ed407864f9cf431ebd8018e1a6862294 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:39:29.725989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.726699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.728255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1121/2247] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:39:29 2022] DELETE /identity/v3/projects/ed407864f9cf431ebd8018e1a6862294 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:29.732412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:29.734212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:29.748314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0c7140-8b18-49a6-b306-a82f404a2090 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:29.817086 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:29.818036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:29.818402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:29.818762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] PATH_INFO: `/v3/users/a684dbd081a044e693e1197f757b6658` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:29.820365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a684dbd081a044e693e1197f757b6658)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:29.822433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:29.877538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a684dbd081a044e693e1197f757b6658'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.885337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a684dbd081a044e693e1197f757b6658'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.900379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a684dbd081a044e693e1197f757b6658'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.901361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a684dbd081a044e693e1197f757b6658'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.902158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0f6f5601-f74d-4b73-ac09-caf4d53a964c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a684dbd081a044e693e1197f757b6658'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:29.904116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:29.921173 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1127/2248] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:39:29 2022] DELETE /identity/v3/users/a684dbd081a044e693e1197f757b6658 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:30.103291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:39:30.104890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:30.122160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3eaefc-9dcf-45a9-adf1-f6979bd16535 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:39:30.206085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:39:30.206773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:30.207051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:30.207321 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] PATH_INFO: `/v3/projects/f2ccb8caf53844bebcbe26010b009e00` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:30.208467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2ccb8caf53844bebcbe26010b009e00)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:39:30.210440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:39:30.247060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:30.263666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2ccb8caf53844bebcbe26010b009e00', 'request_id': 'req-ef74082f-498b-481b-a1dc-3b04e6afd7d9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:30.270830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2ccb8caf53844bebcbe26010b009e00', 'request_id': 'req-ef74082f-498b-481b-a1dc-3b04e6afd7d9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:30.278510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2ccb8caf53844bebcbe26010b009e00', 'request_id': 'req-ef74082f-498b-481b-a1dc-3b04e6afd7d9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:30.291541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] The token cache is being invalidate because project f2ccb8caf53844bebcbe26010b009e00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:39:30.291834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:30.292510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:39:30.294447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1122/2249] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:39:30 2022] DELETE /identity/v3/projects/f2ccb8caf53844bebcbe26010b009e00 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:39:39.919870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:40.288322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:39:43.114965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c05d5f-4db7-49ce-88e8-50e0e26c2813 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:39:43.115507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c05d5f-4db7-49ce-88e8-50e0e26c2813 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:39:43.115507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c05d5f-4db7-49ce-88e8-50e0e26c2813 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:39:43.135974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-09c05d5f-4db7-49ce-88e8-50e0e26c2813 None None] MFA Rules not processed for user `87ca9fc2e013477cad3a3de8e8811274`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:39:43.137362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-09c05d5f-4db7-49ce-88e8-50e0e26c2813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:39:43.212365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1128/2250] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:39:43 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:39:53.205046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=25 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:09.461796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:09.463378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:09.524684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef74082f-498b-481b-a1dc-3b04e6afd7d9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:09.528291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:09.529342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:09.529618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:09.529883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:09.531779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:09.606019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:09.608300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499ec959-b00b-4249-8b59-7e8995793e0d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:09.620515 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1123/2251] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:40:09.718958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:09.719757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:09.719757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:09.743517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] MFA Rules not processed for user `26abb94392364bdc82f6bb508a611d44`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:40:09.744769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:09.843479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1129/2252] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:40:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:40:18.042544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:18.044493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:18.044493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:18.064637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] MFA Rules not processed for user `9c42d5aec138420a90b87226dbe6dc20`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:40:18.065960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:18.131779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1124/2253] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:40:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:40:19.830759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=25 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:24.882772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:24.885661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:24.970023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94bfd9-3930-4285-91ed-cab34832b237 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:24.975198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:24.975987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:24.976365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:24.976734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:24.978777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:24.993570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:24.995190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:25.009277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd72dda2-dd4f-4540-b00f-54d5f506aabc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:25.015195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:25.016802 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:25.017409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:25.017688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:25.019580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:25.067087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:25.068673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc891701-ce8f-42f3-9977-3033e75aa5e5 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:25.072680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:25.082586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1130/2254] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:40:25.094654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:25.096136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:25.107753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1125/2255] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:40:35.074677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=48 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:35.102370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:36.106955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71c601-0147-4ee0-a182-0129c9f191dc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:36.107726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71c601-0147-4ee0-a182-0129c9f191dc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:36.107726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71c601-0147-4ee0-a182-0129c9f191dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:36.129221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0b71c601-0147-4ee0-a182-0129c9f191dc None None] MFA Rules not processed for user `0c1de2efe2ed42508d7ab4a2e9d91e92`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:40:36.130444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b71c601-0147-4ee0-a182-0129c9f191dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:36.137561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1131/2256] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:40:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:40:36.709990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:36.722516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f2200d-39dc-4f70-822b-89115ce96302 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:36.727382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:36.728527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:36.729042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:36.729565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:36.732939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:36.751104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:36.753981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c16d43-d4f3-4e67-a95c-e52b77e2ee25 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:36.769887 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1126/2257] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:40:37.592119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085274fa-1ec2-4d28-8ba5-d8ea1f4b63f9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:37.592119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085274fa-1ec2-4d28-8ba5-d8ea1f4b63f9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:37.593153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085274fa-1ec2-4d28-8ba5-d8ea1f4b63f9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:37.594038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1132/2258] 10.222.0.16 () {54 vars in 961 bytes} [Tue Oct 4 08:40:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:40:37.601933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c179c-bca9-4fd5-868a-ebdb22e649d8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:37.602659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c179c-bca9-4fd5-868a-ebdb22e649d8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:37.603173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c179c-bca9-4fd5-868a-ebdb22e649d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:37.632209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-552c179c-bca9-4fd5-868a-ebdb22e649d8 None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:40:37.633456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-552c179c-bca9-4fd5-868a-ebdb22e649d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:37.755518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1127/2259] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Oct 4 08:40:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:40:37.765593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085274fa-1ec2-4d28-8ba5-d8ea1f4b63f9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:37.783699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085274fa-1ec2-4d28-8ba5-d8ea1f4b63f9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:37.791272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:37.793278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:37.793278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:37.793278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:37.797009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:37.813896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:37.816240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:37.832931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1133/2260] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:40:39.920127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:39.920652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:39.920652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:39.942978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] MFA Rules not processed for user `ab4b3c2b3f4b4978b0eb472f9c2736b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:40:39.944208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:40.011557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1128/2261] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:40:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:40:41.728170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:40:41.743417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b6c1d-291f-4cf6-80b6-97f8317333e2 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:40:41.747425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:40:41.748034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:40:41.748304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:40:41.748570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:40:41.750689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:40:41.761097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:40:41.763167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:40:41.772392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1134/2262] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:40:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:40:50.003743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=60 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:40:51.768326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=15 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:41:01.789228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:41:01.799810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f472ff39-43d3-4fc9-8e2f-07e91b315673 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:41:01.803394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:41:01.803999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:41:01.804274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:41:01.804546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:41:01.806310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:41:01.818342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:41:01.819933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:41:01.829545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1129/2263] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:41:01 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:41:11.825260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=6 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:43:22.060689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:43:22.062250 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:22.150769 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648c2ef3-1e22-4edf-b762-2b8000630555 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:43:22.156258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:43:22.156981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:43:22.156981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:43:22.157257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:43:22.159062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:43:22.172311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:43:22.173889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:43:22.184370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1135/2264] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:43:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:43:32.179695 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=37 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:46:16.747127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:46:16.776217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240ed9a-652d-437b-a5cb-5b4ee5dd3a08 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:46:16.779840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:46:16.780435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:46:16.781184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:46:16.781184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:46:16.789371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:46:16.790842 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:46:16.798905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1130/2265] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:46:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:46:26.795317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=15 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:48:26.057008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:48:26.072104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc71fe-1228-446e-9575-cf4dfebd6092 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:48:26.080848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:48:26.081935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:48:26.082427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:48:26.083077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:48:26.098711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:48:26.101436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:48:26.121842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1136/2266] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:48:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:48:36.113052 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:20.549859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:20.551483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:20.618918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc53a14b-a2eb-4e2d-830c-b1dd2091d79b service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:20.622500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:20.623120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:20.623390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:20.623660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:20.625470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:20.634783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:20.636192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:20.643882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1131/2267] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:51:30.640401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=31 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:51:53.992313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:51:53.993958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:54.084347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dc01f4-b561-41ac-994c-76f4ece0a496 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:51:54.089132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:51:54.089741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:51:54.090020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:51:54.090284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:51:54.092157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:51:54.191066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:51:54.192586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:51:54.216703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1137/2268] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:51:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:52:04.213915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=70 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:52:04.420927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:52:04.430524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418d3466-6e24-40cb-87f9-d55066743a69 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:52:04.434051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:52:04.434672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:52:04.434950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:52:04.435219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:52:04.436911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:52:04.517857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:52:04.519333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:52:04.537864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1132/2269] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:52:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:52:14.535156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=41 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:21.118921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.134634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51461265-1aa8-4b51-bc29-3695619a45b8 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.140318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.141480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.141902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.142303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.145026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.260396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.261929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1212077-a6ac-4a9b-b67a-3f52012a1ab8 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.285682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1138/2270] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:21.293754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.304560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3dd42-7f8b-4d29-a348-6e99b716a8ea service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.308322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.308940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.309223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.309496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.311532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] Could not find project: ed407864f9cf431ebd8018e1a6862294.: keystone.exception.ProjectNotFound: Could not find project: ed407864f9cf431ebd8018e1a6862294. Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 04 08:53:21.326855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:53:21.328927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 04 08:53:21.330344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 04 08:53:21.331880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 04 08:53:21.333253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: ed407864f9cf431ebd8018e1a6862294. Oct 04 08:53:21.333253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Oct 04 08:53:21.333253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1133/2271] 10.222.0.16 () {60 vars in 1396 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 113 bytes in 35 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Oct 04 08:53:21.360549 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.361028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.361443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.433008 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] MFA Rules not processed for user `02d91fccc1c8400392fbbc34416f552b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:21.434268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.444538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1139/2272] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:53:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:21.454877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.464790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cb72a7-f58a-4354-85a2-e33701b6ade3 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.468569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.469219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.469493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.469786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.481630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.483296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.496660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1134/2273] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:21.556132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.557054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.572429 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c505ac6-b876-4311-9d66-310317e128e8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.578421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.579078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.579352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.579623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.581591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.675777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.678064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.683380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:21.701327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1140/2274] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:21.829902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.842592 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036aeed6-38fd-4adc-8f65-2e0406b63ca4 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.848849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.849573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.850064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.850408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.852541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:21.867440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:21.869163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:21.881967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1135/2275] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:21.921228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:21.933668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ad394-5006-4b99-a949-0577ca83064e service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:21.939652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:21.940288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:21.940555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:21.940854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:21.943617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:22.037459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.039306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.062110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1141/2276] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:21 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:22.146023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.183959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9ea73-7d8d-4fde-a2fa-b73e4717c785 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.190275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.191106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.191482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.191853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.193933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:22.232213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.233833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.246432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1136/2277] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:22.642577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:22.696903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d39cc-ef68-4e16-95c4-dfba46df38f4 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:22.705414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:22.706416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:22.706802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:22.707153 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:22.711131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:22.850743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:22.853707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd540911-e55b-4391-821f-3580b9efbd5c service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:22.891635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1142/2278] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:22 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:23.282265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:23.311023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f86210-7c50-4413-a9b8-23175c1d0fbb service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:23.334921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:23.335552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:23.335820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:23.336279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:23.340693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:23.465981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:23.467705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2681371e-1e6c-4817-a3bd-49aaaf8113cc service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:23.492314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1137/2279] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:23 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:29.877197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.877883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.878197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.881311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1143/2280] 10.222.0.16 () {54 vars in 912 bytes} [Tue Oct 4 08:53:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 04 08:53:29.887492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:29.888345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:29.888345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:29.955247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] MFA Rules not processed for user `18e6e8451d334251a87f84b03e57a4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:29.956506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:29.973359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:30.057175 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1138/2281] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:53:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:30.085342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:30.088487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:30.119301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca95891-ff1f-46ae-a4c7-00b867116cde None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:30.128003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:30.129015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:30.129684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:30.130168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:30.144424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:30.147387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:30.155323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:30.169058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1144/2282] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:32.047816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:32.050614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:32.072069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f246b16-fc4e-49a8-b25d-7a21d3e735de None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:32.078894 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:32.079626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:32.079991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:32.080447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:32.083882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:32.097270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:32.099796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:32.111318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1139/2283] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:33.529975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:33.554196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f60616-b659-4101-87dc-4db9a023f1e8 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:33.563942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:33.565540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:33.566264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:33.567039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:33.571119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:33.807004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:33.811541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:33.885207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1145/2284] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:35.686110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:35.770948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6543f655-8e87-40b1-956b-ec41fc18a317 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:35.803389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:35.809333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:35.813236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:35.816695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:35.828328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:35.889067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:35.893650 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:35.929782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1140/2285] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:40.124829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:40.168963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c256-5a57-41e2-9b10-6dc665df9aea service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:40.178713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:40.180152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:40.182070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:40.182629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:40.188034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:40.220934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:40.224637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:40.252311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1146/2286] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:42.467637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:42.491933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ec62f-f8ea-4620-85d9-f5f78ac07957 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:42.498702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:42.499703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:42.500168 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:42.501080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:42.504457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:42.521552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:42.524506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddf5e33-048d-4cac-8c19-d14a4cd95481 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:42.550261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1141/2287] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:43.317192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:43.360961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6948e896-4d7f-4e65-8de8-f995fd8fa473 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:43.367063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:43.367712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:43.368489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:43.369080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:43.374282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:43.413213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:43.415733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:43.446768 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1147/2288] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:45.671664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f2a798-1649-4779-9707-0786c2288383 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:45.674924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f2a798-1649-4779-9707-0786c2288383 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:45.677464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f2a798-1649-4779-9707-0786c2288383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:45.749089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-84f2a798-1649-4779-9707-0786c2288383 None None] MFA Rules not processed for user `78634b34107d49cdbf0126014bf8f6f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:45.752184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84f2a798-1649-4779-9707-0786c2288383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:45.767111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1142/2289] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:46.936853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:46.952584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe128d1-a330-4b8a-97ca-f262faf7aa7d service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:46.987365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:46.991371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:46.992259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:46.992711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:47.007297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:47.081390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:47.086381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:47.124444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1148/2290] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:49.643478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f2a798-1649-4779-9707-0786c2288383 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:49.688444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f2a798-1649-4779-9707-0786c2288383 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:49.708997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:49.710588 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:49.711649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:49.712939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:49.721622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:49.747012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:49.750052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:49.766401 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1143/2291] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:50.067792 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:50.069456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:50.119801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a042963-c848-4b8f-abf8-59c7c46b890e service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:50.137326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:50.137986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:50.138264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:50.138854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] PATH_INFO: `/v3/users/ab4b3c2b3f4b4978b0eb472f9c2736b1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:50.143412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab4b3c2b3f4b4978b0eb472f9c2736b1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:50.156216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:50.412843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab4b3c2b3f4b4978b0eb472f9c2736b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:50.442973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab4b3c2b3f4b4978b0eb472f9c2736b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:50.482405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab4b3c2b3f4b4978b0eb472f9c2736b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:50.483872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab4b3c2b3f4b4978b0eb472f9c2736b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:50.484915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab4b3c2b3f4b4978b0eb472f9c2736b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:50.496923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:50.531867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1149/2292] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:50 2022] DELETE /identity/v3/users/ab4b3c2b3f4b4978b0eb472f9c2736b1 => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:50.881908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:50.915046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0f683-30db-41c8-86f1-0fc2af4cfc1c service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:51.068088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:51.068797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:51.069304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:51.069799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] PATH_INFO: `/v3/projects/2f94e3ae140b43739febe1111e18604b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:51.080271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f94e3ae140b43739febe1111e18604b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:51.086691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:51.225185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:51.276436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f94e3ae140b43739febe1111e18604b', 'request_id': 'req-b4707c83-6f5f-4e66-96fd-0162acf6c799'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.352208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f94e3ae140b43739febe1111e18604b', 'request_id': 'req-b4707c83-6f5f-4e66-96fd-0162acf6c799'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.382945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f94e3ae140b43739febe1111e18604b', 'request_id': 'req-b4707c83-6f5f-4e66-96fd-0162acf6c799'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.418499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] The token cache is being invalidate because project 2f94e3ae140b43739febe1111e18604b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:51.418944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.420148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.427972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1144/2293] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:50 2022] DELETE /identity/v3/projects/2f94e3ae140b43739febe1111e18604b => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:51.428649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:51.431776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:51.467506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63bc2e-bc17-4afb-8198-1dfe157d2d50 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:51.781290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:51.782943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:51.783791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:51.784473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] PATH_INFO: `/v3/users/80c68cf98eea4ceaae245f70ad7a1b8b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:51.831020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=80c68cf98eea4ceaae245f70ad7a1b8b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:51.833431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:51.948855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80c68cf98eea4ceaae245f70ad7a1b8b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:51.968306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80c68cf98eea4ceaae245f70ad7a1b8b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:52.014391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80c68cf98eea4ceaae245f70ad7a1b8b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:52.015224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80c68cf98eea4ceaae245f70ad7a1b8b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:52.016263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80c68cf98eea4ceaae245f70ad7a1b8b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:52.054162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1150/2294] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:51 2022] DELETE /identity/v3/users/80c68cf98eea4ceaae245f70ad7a1b8b => generated 0 bytes in 621 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:52.084470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:52.086729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:52.283062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4707c83-6f5f-4e66-96fd-0162acf6c799 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:52.328268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:52.331107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:52.510054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:52.511142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.511640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.512109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.516228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:52.545994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:52.549192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:52.587348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1145/2295] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:52.765259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548ed3a7-bd49-40c2-bd2c-546ef0a42bbd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:52.790116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:52.791557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:52.792275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:52.792969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] PATH_INFO: `/v3/users/0c1de2efe2ed42508d7ab4a2e9d91e92` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:52.795738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c1de2efe2ed42508d7ab4a2e9d91e92)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:52.802535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:52.844581 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:52.991835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c1de2efe2ed42508d7ab4a2e9d91e92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.012489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c1de2efe2ed42508d7ab4a2e9d91e92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.068799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c1de2efe2ed42508d7ab4a2e9d91e92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.075973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c1de2efe2ed42508d7ab4a2e9d91e92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.077297 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c1de2efe2ed42508d7ab4a2e9d91e92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.086147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:53.087782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:53.111889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1151/2296] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:52 2022] DELETE /identity/v3/users/0c1de2efe2ed42508d7ab4a2e9d91e92 => generated 0 bytes in 784 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:53.122776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d0fd31-714a-4696-b0cd-39207e49a24d service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:53.271292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:53.272704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:53.273376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:53.274170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] PATH_INFO: `/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:53.285573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:53.311228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9f8f3f4a8dc45919e6bc8a78b745a76)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:53.314899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:53.444736 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9f8f3f4a8dc45919e6bc8a78b745a76', 'request_id': 'req-150db2dd-316e-48a4-9010-66277e67b49d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.470607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9f8f3f4a8dc45919e6bc8a78b745a76', 'request_id': 'req-150db2dd-316e-48a4-9010-66277e67b49d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.490509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9f8f3f4a8dc45919e6bc8a78b745a76', 'request_id': 'req-150db2dd-316e-48a4-9010-66277e67b49d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.522748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] The token cache is being invalidate because project b9f8f3f4a8dc45919e6bc8a78b745a76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:53.524012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.526216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-150db2dd-316e-48a4-9010-66277e67b49d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:53.531041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1146/2297] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:53 2022] DELETE /identity/v3/projects/b9f8f3f4a8dc45919e6bc8a78b745a76 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:54.092010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:54.093819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:54.116752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1410f9da-fa0c-4c49-9c48-979c490e6e14 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:54.246273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:54.247092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:54.247643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:54.248023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] PATH_INFO: `/v3/projects/2b940c43c46c402bbc4c3c0461b51061` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:54.249344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b940c43c46c402bbc4c3c0461b51061)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:54.252042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:54.330493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b940c43c46c402bbc4c3c0461b51061', 'request_id': 'req-801b7355-6ce5-496f-a69f-b53b1db46de7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.341436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b940c43c46c402bbc4c3c0461b51061', 'request_id': 'req-801b7355-6ce5-496f-a69f-b53b1db46de7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.355509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b940c43c46c402bbc4c3c0461b51061', 'request_id': 'req-801b7355-6ce5-496f-a69f-b53b1db46de7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.383352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] The token cache is being invalidate because project 2b940c43c46c402bbc4c3c0461b51061 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:54.384752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.387397 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.393887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1152/2298] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:54 2022] DELETE /identity/v3/projects/2b940c43c46c402bbc4c3c0461b51061 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:54.397549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68101186-2825-4e62-a2e4-df8bec351de4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:54.398832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68101186-2825-4e62-a2e4-df8bec351de4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:54.399423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68101186-2825-4e62-a2e4-df8bec351de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:54.463181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:54.467474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:54.507868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:54.546997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-68101186-2825-4e62-a2e4-df8bec351de4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:54.549480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-68101186-2825-4e62-a2e4-df8bec351de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:54.641688 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b7355-6ce5-496f-a69f-b53b1db46de7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:54.656765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:54.657757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:54.658040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:54.658313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] PATH_INFO: `/v3/users/87ca9fc2e013477cad3a3de8e8811274` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:54.660590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] RBAC: Authorizing `identity:delete_user(user_id=87ca9fc2e013477cad3a3de8e8811274)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:54.663194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:54.737410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1147/2299] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:54.740250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:54.740700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:54.741265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:54.777937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:54.780388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:54.808403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1148/2300] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:54.809759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:54.825623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f03807-5b9c-4da7-b56b-6dcb9d072765 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:54.827719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87ca9fc2e013477cad3a3de8e8811274'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.835263 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:54.836098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:54.837163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:54.837163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:54.837804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:54.838252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:54.841014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:54.870401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87ca9fc2e013477cad3a3de8e8811274'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.929499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1149/2301] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:54.933514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:54.942629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87ca9fc2e013477cad3a3de8e8811274'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.943383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87ca9fc2e013477cad3a3de8e8811274'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.944005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87ca9fc2e013477cad3a3de8e8811274'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:54.980369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d550197d-f3d4-4438-9b34-f35e97884e5a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.001922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1153/2302] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:53:54 2022] DELETE /identity/v3/users/87ca9fc2e013477cad3a3de8e8811274 => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:55.087223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:55.137913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.140839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.140839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.141552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.143509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.147221 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.260545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1150/2303] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:54 2022] POST /identity/v3/projects => generated 363 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:55.267918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.272189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.301860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce72518-7e21-4233-bec3-40fc7906b7df admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.371864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.374640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.438405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.439074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.439355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.439625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.440286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.444373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.478617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1154/2304] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:55 2022] POST /identity/v3/users => generated 282 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:55.482659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.497675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05642903-d50b-4b43-b6d2-224c37c19cab admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.500975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13047d3-f013-404d-8e80-d4c4dd67748f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.503410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.505529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.505996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.508169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] PATH_INFO: `/v3/projects/3e938efbd6804a86946d80ae92e88b16` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.509231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e938efbd6804a86946d80ae92e88b16)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.510033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.510717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.511016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.511273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.511930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.512245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.514277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.531383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1155/2305] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:55 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:55.532855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.546631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6553eb0f-9a2b-4147-a632-f2528676cc3e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.554937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.556061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.556061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.556272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.556872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.558778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.567011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:55.579873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1156/2306] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:55 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:53:55.583050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.588239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e938efbd6804a86946d80ae92e88b16', 'request_id': 'req-bfa32fcb-93a8-4320-9e28-f51130e9831a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:55.596776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbb478c-79e6-4341-9afb-32eb77ec4a0d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.598210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e938efbd6804a86946d80ae92e88b16', 'request_id': 'req-bfa32fcb-93a8-4320-9e28-f51130e9831a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:55.610379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e938efbd6804a86946d80ae92e88b16', 'request_id': 'req-bfa32fcb-93a8-4320-9e28-f51130e9831a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:55.641818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] The token cache is being invalidate because project 3e938efbd6804a86946d80ae92e88b16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:53:55.643905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:55.645548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bfa32fcb-93a8-4320-9e28-f51130e9831a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:53:55.648427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1151/2307] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:53:55 2022] DELETE /identity/v3/projects/3e938efbd6804a86946d80ae92e88b16 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:55.656387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b599a36-e042-4e30-badb-3b3bf1c084d4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.656699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b599a36-e042-4e30-badb-3b3bf1c084d4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.656949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b599a36-e042-4e30-badb-3b3bf1c084d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.680793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9b599a36-e042-4e30-badb-3b3bf1c084d4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:55.682090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b599a36-e042-4e30-badb-3b3bf1c084d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.689803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.691550 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.692033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.692489 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] PATH_INFO: `/v3/projects/0845a9d4288c4d91988c5b31ba2ef1f9/users/8d20e41914db4dc89f8efb8f73aec445/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.719366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0845a9d4288c4d91988c5b31ba2ef1f9, user_id=8d20e41914db4dc89f8efb8f73aec445, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.725320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd45556-d252-46ff-9922-1e1ca65bb72e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.745776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:55.748901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1157/2308] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:55 2022] PUT /identity/v3/projects/0845a9d4288c4d91988c5b31ba2ef1f9/users/8d20e41914db4dc89f8efb8f73aec445/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:55.760025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.760025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.760459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.788705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1152/2309] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:55.790984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.791326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.791776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.805155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] MFA Rules not processed for user `8d20e41914db4dc89f8efb8f73aec445`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:55.806604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.816772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:55.819173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.908687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1158/2310] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:55 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:55.911458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1153/2311] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:53:55.918806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.919129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.923306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:55.931750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8934a8b7-0d8e-4746-bff9-66763baee482 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:55.937028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:55.937660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:55.938012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:55.938353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:55.939142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:53:55.940258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:55.941847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:55.963160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1159/2312] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:53:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:53:55.966483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:55.992383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cff235-32e6-4c24-8612-c441fc265155 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.001230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.002028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.002540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.003845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.004806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.009061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.026144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf43b06-9042-48d0-a839-6d45d3fab62c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.030446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.031241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.031623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.032000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.051869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.055179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.076955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1154/2313] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:55 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:56.103748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1160/2314] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:55 2022] POST /identity/v3/projects => generated 365 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:53:56.109599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:56.146525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8455749-bf7e-4df8-ab31-a11406d4dc95 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.241254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:56.244401 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:56.327018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.327351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.327676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.327895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.328735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.330977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.377965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1155/2315] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:56 2022] POST /identity/v3/users => generated 283 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:56.381248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:56.385886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:56.406133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af28f739-7e5a-47ef-aa00-26104cea6adf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.417032 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.418083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.418434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.418778 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.419933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.422594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.424218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5063bc-66de-4a17-9520-04333cfb5908 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.431030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.432092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.432826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.433306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.436187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:56.446945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1156/2316] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:56 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:56.460115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:56.469809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.473989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.485462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefd3bfd-30ed-4b1d-99f2-e9e7d80616d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.497446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.498705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.499997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.499997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.500845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.504775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72556f-d2d9-45b1-b97b-929d21c05587 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.511495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1161/2317] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:53:56.527180 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1157/2318] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:56 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:56.535649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:56.561735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117254d-192e-4521-af67-19f706374a56 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:56.572549 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:56.573999 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.574503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.575051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] PATH_INFO: `/v3/projects/badbfa4855e3450899407cdb55a2853c/users/fcbcad15b2b14d12be6c6d0dd04d9e46/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.610751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] RBAC: Authorizing `identity:create_grant(project_id=badbfa4855e3450899407cdb55a2853c, user_id=fcbcad15b2b14d12be6c6d0dd04d9e46, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:56.618666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:56.675535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1162/2319] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:56 2022] PUT /identity/v3/projects/badbfa4855e3450899407cdb55a2853c/users/fcbcad15b2b14d12be6c6d0dd04d9e46/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:53:56.682540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:56.683069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:56.683538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:56.700172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:56.743063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] MFA Rules not processed for user `fcbcad15b2b14d12be6c6d0dd04d9e46`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:56.744419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:56.850330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1158/2320] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:56.853462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.015193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5977f8-3a2a-4ddb-8468-f4bc5ae78694 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.022881 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.023518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.023793 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.024062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.024736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.026729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.064458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1163/2321] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:53:56 2022] POST /identity/v3/projects => generated 365 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:57.070244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.083421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fd3ed3-270b-4608-9517-a35a6cdae951 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.199428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.200353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.200814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.201096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.201910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.204187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.241167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1159/2322] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:53:57 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:53:57.245892 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.262187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af4bda4-3833-4399-89fb-04908a402d22 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.269108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.269655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.270003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.270336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.271104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.272822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.288618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1164/2323] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:57 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:57.293276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.311976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115f65af-4870-43e5-9086-c0884652c199 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.317579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.318271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.318569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.318844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] PATH_INFO: `/v3/projects/d77b9697064d43ea83816954a926d3eb/users/52e155335e9e4cf3bc722b90be136e49/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.336418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d77b9697064d43ea83816954a926d3eb, user_id=52e155335e9e4cf3bc722b90be136e49, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.339476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.356785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1160/2324] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:57 2022] PUT /identity/v3/projects/d77b9697064d43ea83816954a926d3eb/users/52e155335e9e4cf3bc722b90be136e49/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:57.361556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.374104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3f90d-c502-4b0c-8172-7df4ed909c1d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.461294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.462480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.462958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.463800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.464384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.466149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db071c-2f18-413b-84b5-5eb18a49386f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.482596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1165/2325] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:53:57 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:53:57.488944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.508338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b3ad9-8e06-4628-a236-108b287e35dc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.517428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.518133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.518413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.519316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] PATH_INFO: `/v3/projects/d77b9697064d43ea83816954a926d3eb/users/52e155335e9e4cf3bc722b90be136e49/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.522110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d77b9697064d43ea83816954a926d3eb, user_id=52e155335e9e4cf3bc722b90be136e49, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.524966 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.544463 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1161/2326] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:53:57 2022] PUT /identity/v3/projects/d77b9697064d43ea83816954a926d3eb/users/52e155335e9e4cf3bc722b90be136e49/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:53:57.563342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48752328-84ba-46e2-be10-26338456c23d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.563866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48752328-84ba-46e2-be10-26338456c23d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.564363 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48752328-84ba-46e2-be10-26338456c23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.613719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-48752328-84ba-46e2-be10-26338456c23d None None] MFA Rules not processed for user `52e155335e9e4cf3bc722b90be136e49`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:53:57.616193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-48752328-84ba-46e2-be10-26338456c23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:57.642598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:57.751142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1166/2327] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:53:57 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:53:57.763741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:57.765928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:53:57.830865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:53:57.853979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1d604a-fc81-41dc-a012-9dc3ec9c7709 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:57.866752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:57.867930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:57.868628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:57.872181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:57.889313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:57.892392 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:57.920408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1162/2328] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:57 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:53:58.121377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48752328-84ba-46e2-be10-26338456c23d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:53:58.139132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48752328-84ba-46e2-be10-26338456c23d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:53:58.161546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:53:58.165499 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:53:58.165787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:53:58.166172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:53:58.336189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:53:58.337909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:53:58.354476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1167/2329] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:53:58 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:01.858815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:01.862308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:01.941068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:02.025318 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12c19-4e2a-4ece-88c9-6d1cfaa98a9c service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:02.034797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:02.036881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.037653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.038347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] PATH_INFO: `/v3/users/26abb94392364bdc82f6bb508a611d44` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.040838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] RBAC: Authorizing `identity:delete_user(user_id=26abb94392364bdc82f6bb508a611d44)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.046003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.134190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9671479-fc63-4e6d-bfc7-cf5182ae3368 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:02.156652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:02.159515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.160853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.165510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] PATH_INFO: `/v3/users/fcbcad15b2b14d12be6c6d0dd04d9e46` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.169339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fcbcad15b2b14d12be6c6d0dd04d9e46)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.173129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.179402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26abb94392364bdc82f6bb508a611d44'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.197399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26abb94392364bdc82f6bb508a611d44'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.222249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26abb94392364bdc82f6bb508a611d44'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.223158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26abb94392364bdc82f6bb508a611d44'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.241648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-abe0410d-ef16-4ac6-b83b-2f2fd4b6816e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26abb94392364bdc82f6bb508a611d44'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.258672 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1163/2330] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:01 2022] DELETE /identity/v3/users/26abb94392364bdc82f6bb508a611d44 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:02.286976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fcbcad15b2b14d12be6c6d0dd04d9e46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.294466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:02.312129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fcbcad15b2b14d12be6c6d0dd04d9e46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.336467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fcbcad15b2b14d12be6c6d0dd04d9e46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.337984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fcbcad15b2b14d12be6c6d0dd04d9e46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.338899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fcbcad15b2b14d12be6c6d0dd04d9e46'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.365169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1168/2331] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:01 2022] DELETE /identity/v3/users/fcbcad15b2b14d12be6c6d0dd04d9e46 => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:02.370847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.371911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.372660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.410581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:02.411850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.484624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1164/2332] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:02.499973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:02.502006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.586002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:02.590432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.624243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721a6bc-b9aa-4375-bc4c-248fe5acc752 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:02.625083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3708158-ad09-4c3f-bdef-ce5cc70f53fc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:02.631015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:02.632303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.634243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.634933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.653632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:02.654760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.655262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.655754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] PATH_INFO: `/v3/projects/5934621267af4b838889fcbfb9febaea` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.657928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5934621267af4b838889fcbfb9febaea)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.658404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:02.662091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af2267-f469-4cdb-bbc7-d5b153019d52 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.663428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:02.666206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:02.684677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1169/2333] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:02.775807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5934621267af4b838889fcbfb9febaea', 'request_id': 'req-788d4f86-e122-4a06-b859-856794430833'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.783794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5934621267af4b838889fcbfb9febaea', 'request_id': 'req-788d4f86-e122-4a06-b859-856794430833'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.796046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5934621267af4b838889fcbfb9febaea', 'request_id': 'req-788d4f86-e122-4a06-b859-856794430833'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.810238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] The token cache is being invalidate because project 5934621267af4b838889fcbfb9febaea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:02.810611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.811417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-788d4f86-e122-4a06-b859-856794430833 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:02.813345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1165/2334] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:02 2022] DELETE /identity/v3/projects/5934621267af4b838889fcbfb9febaea => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:02.819941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.821322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.822338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:02.849677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:02.851066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:02.955227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1170/2335] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:02.970006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:02.970492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:02.970986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.009907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:03.011808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:03.036661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1166/2336] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:03.042759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.090432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5bd76-63ce-4ebd-b4c7-c8a085e897c7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.107936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.110050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.110960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.111971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.115941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:03.117981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.122357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.164147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1171/2337] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:03.169703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.182857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94ab4ab-c066-4cac-9aba-0e8faf17196b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.197335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.200377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:03.200597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.201345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.201626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.202076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.202798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.205127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.239733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f116e114-a7f0-4d56-bdf8-f68d8300e6f3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.254943 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.255960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.256457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.257137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] PATH_INFO: `/v3/projects/badbfa4855e3450899407cdb55a2853c` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.259631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] RBAC: Authorizing `identity:delete_project(project_id=badbfa4855e3450899407cdb55a2853c)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.263975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.278975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1167/2338] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:03 2022] POST /identity/v3/projects => generated 375 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:03.281865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.371349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'badbfa4855e3450899407cdb55a2853c', 'request_id': 'req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.385731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'badbfa4855e3450899407cdb55a2853c', 'request_id': 'req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.390835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:03.408708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'badbfa4855e3450899407cdb55a2853c', 'request_id': 'req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.439208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] The token cache is being invalidate because project badbfa4855e3450899407cdb55a2853c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:03.439924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.441136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.443064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1172/2339] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:03 2022] DELETE /identity/v3/projects/badbfa4855e3450899407cdb55a2853c => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:03.448624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f51a1db-8faa-49e5-9574-2b241e0a76a0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.449273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.451351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:03.544703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f1e79d-89bc-41ac-a50f-c4fbefc0f02b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.551454 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.552123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.552394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.552667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] PATH_INFO: `/v3/users/52e155335e9e4cf3bc722b90be136e49` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.553920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=52e155335e9e4cf3bc722b90be136e49)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.564107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.571472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.572497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.572904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.573309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.574271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.577595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.601296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:03.611923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1168/2340] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:03 2022] POST /identity/v3/users => generated 288 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:03.618018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.624126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:03.642332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52e155335e9e4cf3bc722b90be136e49'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.656153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63159cd4-3fce-48df-b734-d436dee65016 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.656870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52e155335e9e4cf3bc722b90be136e49'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.667596 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.669640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.670882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.671483 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.673209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.675658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52e155335e9e4cf3bc722b90be136e49'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.676910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52e155335e9e4cf3bc722b90be136e49'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.677492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.678593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52e155335e9e4cf3bc722b90be136e49'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:03.709250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1169/2341] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:03.715859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.720520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1173/2342] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:03 2022] DELETE /identity/v3/users/52e155335e9e4cf3bc722b90be136e49 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:03.721803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:03.746862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3313e246-d517-4992-a7ca-6ebd678d24cf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.851359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.852458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.852792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.853200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.853947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.855862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec5a04b-ed0b-42e4-ba95-d6915b9fd858 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.873227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1170/2343] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:03 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:03.880890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:03.903390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123ad3b7-0c53-4478-85cb-e1f1a4251f6a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:03.912956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:03.913863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.914697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.915394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] PATH_INFO: `/v3/projects/00fd9ac2b3434dd789ae4d53d93fa80c/users/6b28bc14e2324d10a0ca8ab720bbf1fc/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:03.939732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00fd9ac2b3434dd789ae4d53d93fa80c, user_id=6b28bc14e2324d10a0ca8ab720bbf1fc, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:03.943115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:03.966629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1174/2344] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:03 2022] PUT /identity/v3/projects/00fd9ac2b3434dd789ae4d53d93fa80c/users/6b28bc14e2324d10a0ca8ab720bbf1fc/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:03.976585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:03.978073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:03.978415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.027174 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] MFA Rules not processed for user `6b28bc14e2324d10a0ca8ab720bbf1fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:04.030206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.144543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1171/2345] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:04.214395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.217630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.243823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.246790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.306987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=3,keystone:DELETE=13,keystone:SELECT=84 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:04.331403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052aea0c-b95e-4fd7-8aa3-e695f256ee21 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.335723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.336675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.337375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.337656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.339894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.366060 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298a081d-a5d4-4a87-a1aa-848ff6d025f8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.373997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.374696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.375149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.375417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] PATH_INFO: `/v3/projects/d77b9697064d43ea83816954a926d3eb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.378198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d77b9697064d43ea83816954a926d3eb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.379869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.400751 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:04.437854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.439347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.449389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd77b9697064d43ea83816954a926d3eb', 'request_id': 'req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:04.450652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1175/2346] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:04.451257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.453233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.459537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd77b9697064d43ea83816954a926d3eb', 'request_id': 'req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:04.471582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd77b9697064d43ea83816954a926d3eb', 'request_id': 'req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:04.486400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] The token cache is being invalidate because project d77b9697064d43ea83816954a926d3eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:04.486728 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:04.488155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26cb31f6-6a60-4d01-b4fb-7f2080afedb0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:04.494711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1172/2347] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:04 2022] DELETE /identity/v3/projects/d77b9697064d43ea83816954a926d3eb => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:04.496922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d169b1d-bb78-467f-af3f-9005a411ce54 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.497210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d169b1d-bb78-467f-af3f-9005a411ce54 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.497527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d169b1d-bb78-467f-af3f-9005a411ce54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.521910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9d169b1d-bb78-467f-af3f-9005a411ce54 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:04.523766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9d169b1d-bb78-467f-af3f-9005a411ce54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.556541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934444fa-8d56-4211-84c8-a3a4a398f866 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.621766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1173/2348] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:04.630295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.630652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.630962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.647038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '02d91fccc1c8400392fbbc34416f552b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.648198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.648683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.649058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.651063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.652517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:04.653726 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:04.666713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1174/2349] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:04.669903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.682311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca20f4ee-c26a-470a-8d12-e5ed89e08c40 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.687729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.688181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:04.689104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.689366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.689704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.690582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:04.690952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.692677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.708376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1175/2350] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:54:04.714989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.728794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bd0fbb-e376-4c13-8693-10daee609b06 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.734252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.734933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.735246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.735540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.736149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.738789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.774998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.776592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.777164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1176/2351] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:04 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:04.783450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.787801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1176/2352] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:04.868172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14096e31-29ad-46b2-a47c-6d7be62636d2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.873351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.873984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.874268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.874531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.875195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.877166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:04.888376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:04.910630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1177/2353] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:04 2022] POST /identity/v3/users => generated 293 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:04.916877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:04.951291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d33b68-60f1-4f7d-ba50-c304eaef0b4c service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:04.967335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:04.968441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:04.968926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:04.969396 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:04.971861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:04.977897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.007187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1177/2354] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:04 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:05.009339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.027418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9ae95-debe-45d8-b3d5-9df1dcf69b1e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.066201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.067442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.067923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.068370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.069448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.071720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e509d-3939-4b88-952a-d7258524278b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.088576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1178/2355] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:05 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:05.094296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.110924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8329bf-848a-451f-86c2-ad6602ddea19 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.125094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.128814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.129807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.131135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] PATH_INFO: `/v3/projects/eb9f8ecc0c36402b83a708e35835376a/users/8929f52567014e46b991df5be1f13dc6/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.177932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb9f8ecc0c36402b83a708e35835376a, user_id=8929f52567014e46b991df5be1f13dc6, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.181610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.215636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1178/2356] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:05 2022] PUT /identity/v3/projects/eb9f8ecc0c36402b83a708e35835376a/users/8929f52567014e46b991df5be1f13dc6/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:05.228395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.228683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.228958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.267523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] MFA Rules not processed for user `8929f52567014e46b991df5be1f13dc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:05.269913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.328557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.332617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.428695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1179/2357] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:05.432002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.461411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa09f-98cc-4411-9ef4-3d52af8f3d9e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.470111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.471814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.471814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.472560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] PATH_INFO: `/v3/users/6b28bc14e2324d10a0ca8ab720bbf1fc` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.475804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b28bc14e2324d10a0ca8ab720bbf1fc)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.481467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.545024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c1068-0555-4c86-a2c9-a0e4c969fec7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.552447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.553250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.553632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.554049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.555151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.557986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.574691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b28bc14e2324d10a0ca8ab720bbf1fc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.591917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b28bc14e2324d10a0ca8ab720bbf1fc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.612513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1180/2358] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:05 2022] POST /identity/v3/projects => generated 387 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:05.612513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.616396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.628537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b28bc14e2324d10a0ca8ab720bbf1fc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.629564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b28bc14e2324d10a0ca8ab720bbf1fc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.630216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b28bc14e2324d10a0ca8ab720bbf1fc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.647022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1179/2359] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:05 2022] DELETE /identity/v3/users/6b28bc14e2324d10a0ca8ab720bbf1fc => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:05.647949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.649407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.663874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:05.718049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:05.725662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396739bc-7007-4540-86d9-e600afb5b747 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.756898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45386540-3379-4ee2-b7bd-b669a224a41a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.763939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.764870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.765273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.765648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.766418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.769302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53371176-4a98-4ec4-8cfe-22a3918ec052 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.806069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1180/2360] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:05 2022] POST /identity/v3/users => generated 300 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:05.806069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.806069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.806069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.825045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.825844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.826194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.826537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] PATH_INFO: `/v3/users/9c42d5aec138420a90b87226dbe6dc20` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.828081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c42d5aec138420a90b87226dbe6dc20)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.829608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:05.830413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.831110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.846280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1181/2361] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:05.846280 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.868954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3a56f1-8134-4a3e-85f2-18c544226d1c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:05.875375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:05.876303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:05.876779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:05.877214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:05.878723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:05.880904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:05.901235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1182/2362] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:05 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:05.901235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.904194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:05.910956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c42d5aec138420a90b87226dbe6dc20'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.922947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c42d5aec138420a90b87226dbe6dc20'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.943943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c42d5aec138420a90b87226dbe6dc20'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.944812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c42d5aec138420a90b87226dbe6dc20'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.956924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c42d5aec138420a90b87226dbe6dc20'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:05.969888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1181/2363] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:05 2022] DELETE /identity/v3/users/9c42d5aec138420a90b87226dbe6dc20 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:05.969888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:05.969888 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:06.028954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ac043c-18e6-4484-a38b-b5eecff89377 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.102415 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdeb141-18f2-4154-9d00-309348cab420 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.120738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.123202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.123884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.124653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.125228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] PATH_INFO: `/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31/users/d7aeae2937ad4fa5a4956e07170238bd/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.125650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.126031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.126434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.130914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:06.161754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdfb6805df6e4588b3b186c6cde0eb31, user_id=d7aeae2937ad4fa5a4956e07170238bd, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:06.164669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:06.168211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:06.172466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:06.183971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1182/2364] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:05 2022] PUT /identity/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31/users/d7aeae2937ad4fa5a4956e07170238bd/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:06.188491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:06.196350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:06.207147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1183/2365] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:06.280732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d700b709-ba6d-4be6-b8e0-7dfdc02d95bc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.287812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.287812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.288322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.288496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.289049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:06.291593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:06.306922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1183/2366] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:06 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:06.311541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:06.325576 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f71d89a-4ced-4742-9847-376b6e2df3aa service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.335390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.336307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.336742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.338104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] PATH_INFO: `/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31/users/d7aeae2937ad4fa5a4956e07170238bd/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.344663 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdfb6805df6e4588b3b186c6cde0eb31, user_id=d7aeae2937ad4fa5a4956e07170238bd, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:06.353676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51adbd55-a7e8-4142-ad95-1e0d64e466bb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:06.389444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1184/2367] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:06 2022] PUT /identity/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31/users/d7aeae2937ad4fa5a4956e07170238bd/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:06.407671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:06.413729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:06.415946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.416529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.416844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.457571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212edbac-47c3-4df2-9b8b-4fb31bafb36d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.500334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] MFA Rules not processed for user `d7aeae2937ad4fa5a4956e07170238bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:06.508263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:06.544754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:06.651136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.653506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.654036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.654507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] PATH_INFO: `/v3/projects/f0b762e9e807416b925d8d34bb16fbf0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.656459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0b762e9e807416b925d8d34bb16fbf0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:06.661290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:06.698606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1185/2368] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:06 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:06.712548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:06.720031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:06.785451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0b762e9e807416b925d8d34bb16fbf0', 'request_id': 'req-cbc75a67-c7ec-4072-956d-dbf652030bad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:06.797931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0b762e9e807416b925d8d34bb16fbf0', 'request_id': 'req-cbc75a67-c7ec-4072-956d-dbf652030bad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:06.813305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0b762e9e807416b925d8d34bb16fbf0', 'request_id': 'req-cbc75a67-c7ec-4072-956d-dbf652030bad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:06.834611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] The token cache is being invalidate because project f0b762e9e807416b925d8d34bb16fbf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:06.834849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:06.835856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:06.838407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1184/2369] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:06 2022] DELETE /identity/v3/projects/f0b762e9e807416b925d8d34bb16fbf0 => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:06.844158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a057c69b-544b-41aa-9449-5cd7f919a985 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:06.954530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:06.955772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:06.956960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:06.957679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:06.961733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.017049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.024760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.098148 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.100638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.130645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1186/2370] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:06 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:07.166907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.170966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.186008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:07.218493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:07.279082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc75a67-c7ec-4072-956d-dbf652030bad admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.289570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.290718 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.291214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.291695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] PATH_INFO: `/v3/projects/00fd9ac2b3434dd789ae4d53d93fa80c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.296914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00fd9ac2b3434dd789ae4d53d93fa80c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.300792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.338236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e81f81b-601b-4067-9e17-f79d2353e4a2 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.350488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.351548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.351829 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.352107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] PATH_INFO: `/v3/users/8929f52567014e46b991df5be1f13dc6` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.359601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8929f52567014e46b991df5be1f13dc6)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.361685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.403360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00fd9ac2b3434dd789ae4d53d93fa80c', 'request_id': 'req-f16ea86b-22c1-449d-bc05-e86017d350d0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.423783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00fd9ac2b3434dd789ae4d53d93fa80c', 'request_id': 'req-f16ea86b-22c1-449d-bc05-e86017d350d0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.442853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8929f52567014e46b991df5be1f13dc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.444431 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00fd9ac2b3434dd789ae4d53d93fa80c', 'request_id': 'req-f16ea86b-22c1-449d-bc05-e86017d350d0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.452075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8929f52567014e46b991df5be1f13dc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.492613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] The token cache is being invalidate because project 00fd9ac2b3434dd789ae4d53d93fa80c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:07.493129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.495287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8929f52567014e46b991df5be1f13dc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.496143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8929f52567014e46b991df5be1f13dc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.496992 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f16ea86b-22c1-449d-bc05-e86017d350d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.498015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-42feb8be-0ea3-4bbe-aacd-ed11cc11fd54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8929f52567014e46b991df5be1f13dc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.503900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1185/2371] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:07 2022] DELETE /identity/v3/projects/00fd9ac2b3434dd789ae4d53d93fa80c => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:07.513202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1deeeb1-d343-47e0-a409-2b7a84330859 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.513653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1deeeb1-d343-47e0-a409-2b7a84330859 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.513929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1deeeb1-d343-47e0-a409-2b7a84330859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.531552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1187/2372] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:07 2022] DELETE /identity/v3/users/8929f52567014e46b991df5be1f13dc6 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:07.538068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.538068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.538068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.558455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b1deeeb1-d343-47e0-a409-2b7a84330859 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:07.559810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:07.561082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.562194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b1deeeb1-d343-47e0-a409-2b7a84330859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.645541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1188/2373] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:07.667311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1186/2374] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:07.670306 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.671945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.683493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.685156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.686730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.733022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:07.744187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:07.748514 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.762187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d815b356-c3c7-4575-8ba4-4913dc8e23b1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.770483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.770483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.770483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.770483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.772037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1187/2375] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:07.773877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.776372 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:07.789492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.793501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.804553 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d73fd4-cea6-4061-ada7-a15f18d39e29 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.817131 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.817991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1189/2376] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:07.818533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.819192 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.819779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] PATH_INFO: `/v3/users/8d20e41914db4dc89f8efb8f73aec445` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.821481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d20e41914db4dc89f8efb8f73aec445)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.825178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.833142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.875233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df26640-e267-4662-bf2c-602c21748873 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.889138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.891403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.891976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.892672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.894618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:07.895162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.899971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.935372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1190/2377] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:07.937225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d20e41914db4dc89f8efb8f73aec445'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.937897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:07.943336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:07.952758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d20e41914db4dc89f8efb8f73aec445'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.961340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75547f88-fc8b-4c76-995c-c44c2bb07ea6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:07.974070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:07.975307 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:07.975801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:07.976465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:07.977644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:07.982434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:07.992358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d20e41914db4dc89f8efb8f73aec445'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:07.993143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d20e41914db4dc89f8efb8f73aec445'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.017810 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ca0ddf62-9e90-4586-bd41-d2b796c21351 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d20e41914db4dc89f8efb8f73aec445'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.032348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1188/2378] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:07 2022] DELETE /identity/v3/users/8d20e41914db4dc89f8efb8f73aec445 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:08.040698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.041076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.041471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.044931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1191/2379] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:07 2022] POST /identity/v3/projects => generated 403 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:08.052980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.056130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.100027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:08.101369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.234455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a37ef4-7a26-44c1-b9ca-663f440d13e4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.246339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.247192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.247972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.248110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.249210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.252677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.266491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1189/2380] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:08.283821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.288652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.297014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1192/2381] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:08 2022] POST /identity/v3/users => generated 302 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:08.304644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.350236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1155c2a4-b9b3-446c-bd00-deda5348ee69 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.363182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.365159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.365778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.366914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.367998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.371026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.395228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1193/2382] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:08.395228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.396527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.414064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658d1bc3-6693-47d7-afdb-2a5fd4782dcd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.420289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.421038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.422076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.422617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] PATH_INFO: `/v3/projects/eb9f8ecc0c36402b83a708e35835376a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.426379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb9f8ecc0c36402b83a708e35835376a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.429328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.443153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf6866b-b15e-4758-9f7c-92d36ec2ab17 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.455290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.456411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.456856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.457393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.475781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.477310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.494535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1190/2383] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:08.495775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.507485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb9f8ecc0c36402b83a708e35835376a', 'request_id': 'req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.509798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6bd73-a333-431e-9b79-a5389e70a501 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.515803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb9f8ecc0c36402b83a708e35835376a', 'request_id': 'req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.528409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb9f8ecc0c36402b83a708e35835376a', 'request_id': 'req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.544165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] The token cache is being invalidate because project eb9f8ecc0c36402b83a708e35835376a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:08.544530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.545166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.547403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1194/2384] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:08 2022] DELETE /identity/v3/projects/eb9f8ecc0c36402b83a708e35835376a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:08.553795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.555575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.562668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:08.636162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.637547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.637973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.638347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.639162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.641831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.656472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1191/2385] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:08 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:08.661974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.664271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.682336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c65ece-f123-4f0f-ab59-b8c4ee15093d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.694231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.695167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.695539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.695874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] PATH_INFO: `/v3/projects/cfac5d4bfbf642e4aaf96e487730e950/users/be7e29960ad14785af31ee95d44e5a3c/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.700716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:08.709045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfa1df-eeeb-443f-97c1-c48f6fcdc634 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:08.719413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:08.720504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.721000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.722093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] PATH_INFO: `/v3/users/d7aeae2937ad4fa5a4956e07170238bd` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.730372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7aeae2937ad4fa5a4956e07170238bd)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.733980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.747117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfac5d4bfbf642e4aaf96e487730e950, user_id=be7e29960ad14785af31ee95d44e5a3c, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:08.752737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00968671-f128-4082-99fa-5d2869230d14 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:08.799313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1192/2386] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:08 2022] PUT /identity/v3/projects/cfac5d4bfbf642e4aaf96e487730e950/users/be7e29960ad14785af31ee95d44e5a3c/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:08.808383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:08.809146 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:08.812915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:08.841599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7aeae2937ad4fa5a4956e07170238bd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.855908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:08.857696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7aeae2937ad4fa5a4956e07170238bd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.857933 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:08.890239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7aeae2937ad4fa5a4956e07170238bd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.892004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7aeae2937ad4fa5a4956e07170238bd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.892834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7aeae2937ad4fa5a4956e07170238bd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:08.922996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1195/2387] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:08 2022] DELETE /identity/v3/users/d7aeae2937ad4fa5a4956e07170238bd => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:08.970429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1193/2388] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:08.989692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:08.991488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.031365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.036118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.077458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdce132-e6eb-401a-9cc1-a47f0f02b350 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.162571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ca4b-8b1e-4298-849e-15f4f019c029 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.169324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.170381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.170899 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.171389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.177064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.177997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.178383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.178906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] PATH_INFO: `/v3/projects/0845a9d4288c4d91988c5b31ba2ef1f9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.180170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] RBAC: Authorizing `identity:delete_project(project_id=0845a9d4288c4d91988c5b31ba2ef1f9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.182253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.190009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.193488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb84510-7a7e-44eb-be6a-994728b4a7e1 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.214748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1196/2389] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:09.252106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0845a9d4288c4d91988c5b31ba2ef1f9', 'request_id': 'req-039022e5-c8d7-421d-a47b-f453d4db3cba'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.260421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0845a9d4288c4d91988c5b31ba2ef1f9', 'request_id': 'req-039022e5-c8d7-421d-a47b-f453d4db3cba'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.272752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0845a9d4288c4d91988c5b31ba2ef1f9', 'request_id': 'req-039022e5-c8d7-421d-a47b-f453d4db3cba'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.286406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] The token cache is being invalidate because project 0845a9d4288c4d91988c5b31ba2ef1f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:09.286734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.287611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-039022e5-c8d7-421d-a47b-f453d4db3cba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.289454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1194/2390] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:09 2022] DELETE /identity/v3/projects/0845a9d4288c4d91988c5b31ba2ef1f9 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:09.294259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.294914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.294914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.317770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:09.319147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.349045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:09.430249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1197/2391] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:09.434348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.434943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.435523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.447161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.449776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.483056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d0c45-3e92-40bc-9729-ce8cc6b183a0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.492834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:09.494015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.495136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.495223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.495590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.496053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] PATH_INFO: `/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.498882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdfb6805df6e4588b3b186c6cde0eb31)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.503210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.513029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1195/2392] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:09.519399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.535308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0667f-6c18-433a-a8e9-6e50c3b0b909 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.546875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.547903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.548400 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.548714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.549957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:09.550455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.553898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.556859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:09.579631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1196/2393] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:09.586245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.586923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdfb6805df6e4588b3b186c6cde0eb31', 'request_id': 'req-7960516b-0806-4839-a77b-0563b99cc136'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.597031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdfb6805df6e4588b3b186c6cde0eb31', 'request_id': 'req-7960516b-0806-4839-a77b-0563b99cc136'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.605519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd3ae67-691f-4cbe-8425-75e83f8a0384 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.617796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdfb6805df6e4588b3b186c6cde0eb31', 'request_id': 'req-7960516b-0806-4839-a77b-0563b99cc136'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.632983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] The token cache is being invalidate because project fdfb6805df6e4588b3b186c6cde0eb31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:09.633450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.634235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7960516b-0806-4839-a77b-0563b99cc136 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:09.636707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1198/2394] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:09 2022] DELETE /identity/v3/projects/fdfb6805df6e4588b3b186c6cde0eb31 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:09.644868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6650f7-839a-4321-b298-d55c53c006ec None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.645318 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6650f7-839a-4321-b298-d55c53c006ec None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.645759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6650f7-839a-4321-b298-d55c53c006ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.681065 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cf6650f7-839a-4321-b298-d55c53c006ec None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:09.683268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf6650f7-839a-4321-b298-d55c53c006ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.718052 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.718967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.719387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.719764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.720968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.734521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.779488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1197/2395] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/projects => generated 405 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:09.780772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.783447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.802253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1199/2396] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:09.812858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.813238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.813590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.865681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:09.868124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:09.917429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32f8958-dea1-4701-9fd1-26b4256128a9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.927014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.928187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.928187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.928187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.928828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:09.930856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:09.956177 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:09.968964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1198/2397] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/users => generated 303 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:09.976685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.988457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1200/2398] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:09.989604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1108c9-974b-4736-8a4d-92e6e464b9fb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:09.992797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:09.996907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:09.997542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:09.997824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:09.998096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:09.998789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.000525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.010752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319278c7-8a57-4b53-8eeb-7607a2401572 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.016300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1199/2399] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:09 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:10.020924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.022182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.023760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.024243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.024746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.026088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:10.027375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.029842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.044246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98ea69e-13db-4de0-9b4c-7ba7495f8db5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.049374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1201/2400] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:54:10.057289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.059952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.060954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.061414 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.062066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.063322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.067095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.077617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5245dcb2-6bcd-438f-b422-657797b0d8ae admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.083284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1200/2401] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:10 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:10.085618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.086693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.087262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.087698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.088210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.089451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.092710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.100286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d48faf-d768-48ae-b5b8-998aed53dae9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.107238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.108025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.108495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.108732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] PATH_INFO: `/v3/projects/fd62019b03d94f82ad3749682df7fa29/users/88a7503a7165408baee20331113b1605/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.130610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd62019b03d94f82ad3749682df7fa29, user_id=88a7503a7165408baee20331113b1605, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.134391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eef2b9-295d-4dda-bb22-4fc209378164 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.146282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1202/2402] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/projects => generated 387 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:10.155452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.168411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1201/2403] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:10 2022] PUT /identity/v3/projects/fd62019b03d94f82ad3749682df7fa29/users/88a7503a7165408baee20331113b1605/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:10.179644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d9dbd-bef8-4045-a26e-c6e2d1627393 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.179990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d9dbd-bef8-4045-a26e-c6e2d1627393 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.180238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3d9dbd-bef8-4045-a26e-c6e2d1627393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.216387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9d3d9dbd-bef8-4045-a26e-c6e2d1627393 None None] MFA Rules not processed for user `88a7503a7165408baee20331113b1605`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:10.218455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9d3d9dbd-bef8-4045-a26e-c6e2d1627393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:10.240623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:10.285642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccd874-234e-4965-b99e-3fcc71442144 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.357622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1202/2404] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:10.359127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.359411 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.359685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.372143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.373103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.373272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.373451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.374066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.376124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.407960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] MFA Rules not processed for user `be7e29960ad14785af31ee95d44e5a3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:10.409543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:10.414181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1203/2405] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/users => generated 294 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:10.415189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.427042 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2997a11e-b796-425a-bef8-e8ecc1831492 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.434110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.434766 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.435040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.435740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.435946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.438399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.482573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1204/2406] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/projects => generated 403 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:10.483778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.509550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1203/2407] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:10.510390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.525377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0c27ab-dbdd-4d0d-8233-c62a127c4569 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.546967 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:10.603555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9631e21a-0de8-4647-96e2-d4cd3325eebd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.609513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.610000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.610232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.610701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.610976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.611187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.611573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.612422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.612623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.613478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.614939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.616687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.634281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1205/2408] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:10 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:10.634281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.638208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:10.650780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1204/2409] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/users => generated 308 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:10.651683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.661057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c11e25-9e61-46f2-ad55-e84a54718496 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.668468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.669530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.670020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.670399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.671309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.676448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdb5f3-e125-434c-91a9-a80a58ea4ed4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.676685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.681500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.682137 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.682659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.682967 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.683678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.685406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.696705 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1205/2410] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:10 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:10.697607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.709567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d00e3-a6d0-4cc6-bddf-bfe3485ac766 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.715357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.716126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.716444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.716745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.717443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.719226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.730509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1206/2411] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/projects => generated 401 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:10.731422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.738367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1206/2412] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:10 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:10.738367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.751091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cde42-1dc6-4a40-a2c2-888482be367c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.758534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:10.826850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.827388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.827604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6ac8bd-835f-469d-90d4-df25a92572ed admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.827729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.827921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.828577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.830532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.833731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.834411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.834711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.834982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] PATH_INFO: `/v3/projects/a5ee0ef889574572b3cd6a415908d254/users/e15faf90f44d47c3a962ad054dd7f907/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.851512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5ee0ef889574572b3cd6a415908d254, user_id=e15faf90f44d47c3a962ad054dd7f907, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.853707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1207/2413] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/users => generated 307 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:10.854132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.854665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.865664 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825a7eee-c83d-42fb-be78-6241c7c325a4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.868992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1207/2414] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:10 2022] PUT /identity/v3/projects/a5ee0ef889574572b3cd6a415908d254/users/e15faf90f44d47c3a962ad054dd7f907/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:10.869802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.926268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.927000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.927281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.927555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] PATH_INFO: `/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65/users/6419c608db5245ad8f852de8f8af13e3/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.939765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780479b-a037-4da2-a81a-f86e81593ef6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:10.943911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d87192cfe5fc41fbba72dd266a0c8e65, user_id=6419c608db5245ad8f852de8f8af13e3, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.945137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:10.945747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.946025 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.946296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.947265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:10.947448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.948623 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da287c5-a4f2-4365-8705-ea072cc36a48 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:10.960667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1208/2415] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:10 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:10.962219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:10.962347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1208/2416] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:10 2022] PUT /identity/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65/users/6419c608db5245ad8f852de8f8af13e3/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:10.964819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:10.965255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:10.965476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:10.996473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] MFA Rules not processed for user `e15faf90f44d47c3a962ad054dd7f907`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:10.997924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:11.013055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.032544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a7f5-fbd5-4389-88f3-be7c8015b661 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.037449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.038100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.038375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.038679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] PATH_INFO: `/v3/projects/7f675af5f4354320a905f217fdc3c54a/users/3b3016d2e5a54c2ab25c45ed04034df1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.054113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f675af5f4354320a905f217fdc3c54a, user_id=3b3016d2e5a54c2ab25c45ed04034df1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.056780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.068302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1209/2417] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:11.069140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.077748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1209/2418] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:10 2022] PUT /identity/v3/projects/7f675af5f4354320a905f217fdc3c54a/users/3b3016d2e5a54c2ab25c45ed04034df1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:11.078704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.082357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e7527a-94aa-4a6f-8cf5-f6b17b16867d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.085070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.146241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.147068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.147350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.147544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fbf81e-6876-4c12-becc-959436fac52d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.147659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.148268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.150022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.152519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.153073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.153348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.153616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.154251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.156505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.161580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1210/2419] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:11.162417 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.173757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8b4e25-62f2-4a26-860e-767c5d007f97 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.179099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.179697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.179970 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.180235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.180867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.182450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.186473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1210/2420] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:11 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:11.187128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.193191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1211/2421] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:11.194141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.206693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0951d3-9942-42fe-8ba8-9387ddd243c5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.259601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e759606-ef2e-451d-a0c1-56ff8571646c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.264424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.265118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.265394 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.265694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] PATH_INFO: `/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65/users/6419c608db5245ad8f852de8f8af13e3/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.267693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d87192cfe5fc41fbba72dd266a0c8e65, user_id=6419c608db5245ad8f852de8f8af13e3, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.267817 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.269249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.269659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.269934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.270621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.270937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.272760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabcdfd5-f53a-4704-8e3b-2ad7d41203da admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.281758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1211/2422] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:11 2022] PUT /identity/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65/users/6419c608db5245ad8f852de8f8af13e3/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:11.282457 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.296557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1212/2423] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:11 2022] POST /identity/v3/users => generated 299 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:11.297831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.298135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.298352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.328059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] MFA Rules not processed for user `6419c608db5245ad8f852de8f8af13e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:11.329249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:11.346938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.351363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707b8121-3fde-4b51-87a7-9f320f943ba3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.356250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.356904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.357178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.357474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] PATH_INFO: `/v3/projects/7f675af5f4354320a905f217fdc3c54a/users/3b3016d2e5a54c2ab25c45ed04034df1/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.359487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f675af5f4354320a905f217fdc3c54a, user_id=3b3016d2e5a54c2ab25c45ed04034df1, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.362184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.372813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1212/2424] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:11 2022] PUT /identity/v3/projects/7f675af5f4354320a905f217fdc3c54a/users/3b3016d2e5a54c2ab25c45ed04034df1/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:11.373622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.396503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.414414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1213/2425] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:11 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:11.427323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.428778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:11.444309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412d5f73-7a0e-4f49-830d-255be5c32658 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.449243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.449950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.450196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.450523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.451429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.453055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.465782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1213/2426] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:11.470098 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.481769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33dc5e9-73d0-430e-a4b7-6e51124dfd7b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.486525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.487339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.487629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.487960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] PATH_INFO: `/v3/projects/edb31accb7ce4043851e014c7dcf2049/users/94e3d3db176d4a66b279bad5a33ace98/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.504487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82e84c5-675a-4677-a95d-f8f9c61d734c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.509376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=edb31accb7ce4043851e014c7dcf2049, user_id=94e3d3db176d4a66b279bad5a33ace98, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.512632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.530332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1214/2427] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:11 2022] PUT /identity/v3/projects/edb31accb7ce4043851e014c7dcf2049/users/94e3d3db176d4a66b279bad5a33ace98/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:11.534295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.569135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.569846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.570383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.570750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.619960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf602707-9d9d-470e-8844-2dfbf48414ba admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.625050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.625671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.626088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.626367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.627046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.629433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.640032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.640256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1215/2428] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:11.641425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6e290c-2a40-40fe-b68e-4c5def7e7e5e service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.644907 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.653948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1214/2429] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:11.655503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d233d5f-fbd2-4ad5-b3d0-f3ba44ce9593 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.662348 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.663112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.663390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.663662 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] PATH_INFO: `/v3/projects/edb31accb7ce4043851e014c7dcf2049/users/94e3d3db176d4a66b279bad5a33ace98/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.667314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=edb31accb7ce4043851e014c7dcf2049, user_id=94e3d3db176d4a66b279bad5a33ace98, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.669959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.683156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1216/2430] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:11 2022] PUT /identity/v3/projects/edb31accb7ce4043851e014c7dcf2049/users/94e3d3db176d4a66b279bad5a33ace98/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:11.694582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.694582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.694773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.764508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] MFA Rules not processed for user `94e3d3db176d4a66b279bad5a33ace98`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:11.765652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:11.781159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.839217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.876124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1215/2431] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:11 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:11.888106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:11.929580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2788868-0327-458c-8525-01ca83c4cdb2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.935257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.935903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.936214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.936487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] PATH_INFO: `/v3/users/88a7503a7165408baee20331113b1605` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.937655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88a7503a7165408baee20331113b1605)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.939688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.956253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:11.965405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef266b6-c344-4b7f-80b7-595d5194fad7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:11.970941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:11.972270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:11.972625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:11.972917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:11.984303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:11.985758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:11.997748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1216/2432] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:11 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:12.020481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88a7503a7165408baee20331113b1605'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.032425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88a7503a7165408baee20331113b1605'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.068523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88a7503a7165408baee20331113b1605'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.070306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88a7503a7165408baee20331113b1605'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.083746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:12.087310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3bf0e3-f5ce-4b5c-8852-b021732f9438 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88a7503a7165408baee20331113b1605'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.096208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1217/2433] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:11 2022] DELETE /identity/v3/users/88a7503a7165408baee20331113b1605 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:12.099415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf110e8-1040-4c89-af8f-8376a0edf71c service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:12.102655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841eb24-49c4-45a0-9b53-3e72691a2002 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.102655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841eb24-49c4-45a0-9b53-3e72691a2002 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.102655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841eb24-49c4-45a0-9b53-3e72691a2002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.133377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4841eb24-49c4-45a0-9b53-3e72691a2002 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:12.134975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4841eb24-49c4-45a0-9b53-3e72691a2002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:12.203166 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:12.203975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.204320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.205139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] PATH_INFO: `/v3/users/e15faf90f44d47c3a962ad054dd7f907` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.206440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e15faf90f44d47c3a962ad054dd7f907)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:12.208520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:12.220733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1218/2434] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:12.223254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776f17f-c412-4d3f-bd34-0ffaa98d04b6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.223720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776f17f-c412-4d3f-bd34-0ffaa98d04b6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.224212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776f17f-c412-4d3f-bd34-0ffaa98d04b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.266475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9776f17f-c412-4d3f-bd34-0ffaa98d04b6 None None] MFA Rules not processed for user `3b3016d2e5a54c2ab25c45ed04034df1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:12.267829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9776f17f-c412-4d3f-bd34-0ffaa98d04b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:12.282966 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e15faf90f44d47c3a962ad054dd7f907'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.286810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:12.297154 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e15faf90f44d47c3a962ad054dd7f907'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.317648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e15faf90f44d47c3a962ad054dd7f907'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.318874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e15faf90f44d47c3a962ad054dd7f907'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.319721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e15faf90f44d47c3a962ad054dd7f907'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:12.345483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1217/2435] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:12 2022] DELETE /identity/v3/users/e15faf90f44d47c3a962ad054dd7f907 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:12.348232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:12.351073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:12.363148 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1219/2436] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:12 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:12.364972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.365319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.366609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.397752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:12.399110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:12.415930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:12.443275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c0a6ab-040d-43f8-8d99-0659ced9e36a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:12.448487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:12.449610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.449894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.450609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.452136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:12.499917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1220/2437] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:12.511243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:12.523228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a14c97-8f8d-4704-9ed4-d9a79ff80ad0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:12.527240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:12.528090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:12.528442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:12.528784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:12.534784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:12.536683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:12.541696 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:12.543433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:12.547856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1218/2438] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:12.555087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1221/2439] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:12.997639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.000607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.027017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb61-23a2-45bc-bec7-3631539c5577 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:13.047872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:13.047872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:13.047872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:13.047872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] PATH_INFO: `/v3/projects/a5ee0ef889574572b3cd6a415908d254` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:13.053016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5ee0ef889574572b3cd6a415908d254)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:13.054440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:13.116082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.119374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.141766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f55c21-02a9-4b02-8e2c-811e36182b9f service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:13.144329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5ee0ef889574572b3cd6a415908d254', 'request_id': 'req-47b92761-1312-49fe-91f1-feda96d367be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.159218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5ee0ef889574572b3cd6a415908d254', 'request_id': 'req-47b92761-1312-49fe-91f1-feda96d367be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.169651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5ee0ef889574572b3cd6a415908d254', 'request_id': 'req-47b92761-1312-49fe-91f1-feda96d367be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.180683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:13.186864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] The token cache is being invalidate because project a5ee0ef889574572b3cd6a415908d254 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:13.187288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.188510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.192658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1219/2440] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:12 2022] DELETE /identity/v3/projects/a5ee0ef889574572b3cd6a415908d254 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:13.196621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.199573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.230524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:13.231270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:13.231615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:13.231988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] PATH_INFO: `/v3/projects/fd62019b03d94f82ad3749682df7fa29` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:13.233185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd62019b03d94f82ad3749682df7fa29)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:13.235283 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:13.293238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b92761-1312-49fe-91f1-feda96d367be admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:13.298819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd62019b03d94f82ad3749682df7fa29', 'request_id': 'req-937f8a7c-edc5-49ff-871b-cbdf89485105'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.310678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd62019b03d94f82ad3749682df7fa29', 'request_id': 'req-937f8a7c-edc5-49ff-871b-cbdf89485105'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.322972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd62019b03d94f82ad3749682df7fa29', 'request_id': 'req-937f8a7c-edc5-49ff-871b-cbdf89485105'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.345545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] The token cache is being invalidate because project fd62019b03d94f82ad3749682df7fa29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:13.345838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.346823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.348654 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1222/2441] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:13 2022] DELETE /identity/v3/projects/fd62019b03d94f82ad3749682df7fa29 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:13.355667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.357313 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.407642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:13.408337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:13.410668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:13.410668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] PATH_INFO: `/v3/users/94e3d3db176d4a66b279bad5a33ace98` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:13.411404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94e3d3db176d4a66b279bad5a33ace98)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:13.413367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:13.426256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:13.462804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937f8a7c-edc5-49ff-871b-cbdf89485105 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:13.468506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:13.469453 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:13.469749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:13.470063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] PATH_INFO: `/v3/users/6419c608db5245ad8f852de8f8af13e3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:13.471538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6419c608db5245ad8f852de8f8af13e3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:13.473764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:13.496435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94e3d3db176d4a66b279bad5a33ace98'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.506883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94e3d3db176d4a66b279bad5a33ace98'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.532309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94e3d3db176d4a66b279bad5a33ace98'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.533062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94e3d3db176d4a66b279bad5a33ace98'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.533724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94e3d3db176d4a66b279bad5a33ace98'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.540447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6419c608db5245ad8f852de8f8af13e3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.552506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6419c608db5245ad8f852de8f8af13e3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.553055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1220/2442] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:13 2022] DELETE /identity/v3/users/94e3d3db176d4a66b279bad5a33ace98 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:13.579245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6419c608db5245ad8f852de8f8af13e3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.580097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6419c608db5245ad8f852de8f8af13e3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.591466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6419c608db5245ad8f852de8f8af13e3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:13.606382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1223/2443] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:13 2022] DELETE /identity/v3/users/6419c608db5245ad8f852de8f8af13e3 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:13.750009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.751704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.791981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:13.872834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef97873-78a2-4929-b194-7a99f83943f6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:13.955390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:13.956935 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:13.957440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:13.957864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] PATH_INFO: `/v3/users/be7e29960ad14785af31ee95d44e5a3c` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:13.959380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be7e29960ad14785af31ee95d44e5a3c)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:13.961591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:13.977287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:13.978783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:13.996749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b09e51-f894-480c-889f-75bf2bf5f9a5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.005812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.016429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.016803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.017155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] PATH_INFO: `/v3/projects/edb31accb7ce4043851e014c7dcf2049` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.019189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=edb31accb7ce4043851e014c7dcf2049)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.021413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.046304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be7e29960ad14785af31ee95d44e5a3c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.057361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be7e29960ad14785af31ee95d44e5a3c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.095730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be7e29960ad14785af31ee95d44e5a3c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.096778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be7e29960ad14785af31ee95d44e5a3c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.097457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be7e29960ad14785af31ee95d44e5a3c'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.105788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'edb31accb7ce4043851e014c7dcf2049', 'request_id': 'req-ed1374c3-5268-425e-8949-ff2f61caba0a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.113777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'edb31accb7ce4043851e014c7dcf2049', 'request_id': 'req-ed1374c3-5268-425e-8949-ff2f61caba0a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.117127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1221/2444] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:13 2022] DELETE /identity/v3/users/be7e29960ad14785af31ee95d44e5a3c => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:14.118198 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.119638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.124328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'edb31accb7ce4043851e014c7dcf2049', 'request_id': 'req-ed1374c3-5268-425e-8949-ff2f61caba0a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.140246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] The token cache is being invalidate because project edb31accb7ce4043851e014c7dcf2049 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:14.140913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.141849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed1374c3-5268-425e-8949-ff2f61caba0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.144559 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1224/2445] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:13 2022] DELETE /identity/v3/projects/edb31accb7ce4043851e014c7dcf2049 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:14.150486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca556c-b852-4305-9423-73476e68c951 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.150948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca556c-b852-4305-9423-73476e68c951 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.151363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca556c-b852-4305-9423-73476e68c951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.181735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9bca556c-b852-4305-9423-73476e68c951 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:14.183564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9bca556c-b852-4305-9423-73476e68c951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.214077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:14.230389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864ec5a4-0232-4f77-b0e3-bec7dac9f503 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.234056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:14.245701 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.246852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.247125 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.247393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] PATH_INFO: `/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.248912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d87192cfe5fc41fbba72dd266a0c8e65)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.251188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.277255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1225/2446] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:14.280838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed40c64-e8b6-401a-9871-6ab1847b0527 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.281016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed40c64-e8b6-401a-9871-6ab1847b0527 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.281374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed40c64-e8b6-401a-9871-6ab1847b0527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.307214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5ed40c64-e8b6-401a-9871-6ab1847b0527 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:14.308555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5ed40c64-e8b6-401a-9871-6ab1847b0527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.319821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd87192cfe5fc41fbba72dd266a0c8e65', 'request_id': 'req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.327855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd87192cfe5fc41fbba72dd266a0c8e65', 'request_id': 'req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.338633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd87192cfe5fc41fbba72dd266a0c8e65', 'request_id': 'req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.365863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] The token cache is being invalidate because project d87192cfe5fc41fbba72dd266a0c8e65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:14.366820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.368577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.371117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1222/2447] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:14 2022] DELETE /identity/v3/projects/d87192cfe5fc41fbba72dd266a0c8e65 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:14.372292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.373709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.398807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1226/2448] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:14.404082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.404082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.404082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.427830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:14.429068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.437867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1227/2449] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:14.439421 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.452806 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5773cb57-8822-4f8e-b087-4564a25466a5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.458767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.459382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.459660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.460058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.461858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:14.461858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.462494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44f601-4539-451a-afcc-7b0fbad24499 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.465339 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2ae6fd-c9be-45f7-bd39-0d43d674518e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.470321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.470997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.471311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.471702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] PATH_INFO: `/v3/projects/cfac5d4bfbf642e4aaf96e487730e950` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.472886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfac5d4bfbf642e4aaf96e487730e950)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.474869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.475598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1228/2450] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:14.476711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.476996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.477268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.497827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:14.499085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.509941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1229/2451] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:14.511119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.523082 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d21ba4-8001-493e-8b48-b80d3dcb551c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.532323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfac5d4bfbf642e4aaf96e487730e950', 'request_id': 'req-d5fb49b9-6893-4d89-8271-6820379a7d04'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.541192 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfac5d4bfbf642e4aaf96e487730e950', 'request_id': 'req-d5fb49b9-6893-4d89-8271-6820379a7d04'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.551906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfac5d4bfbf642e4aaf96e487730e950', 'request_id': 'req-d5fb49b9-6893-4d89-8271-6820379a7d04'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.565254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] The token cache is being invalidate because project cfac5d4bfbf642e4aaf96e487730e950 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:14.565590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.566129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.567835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1223/2452] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:14 2022] DELETE /identity/v3/projects/cfac5d4bfbf642e4aaf96e487730e950 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:14.568627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.570729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.587780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:14.590099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.590945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.591211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.591475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.592103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.594017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.626379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1230/2453] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/projects => generated 383 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:14.627298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.631429 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.656496 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fb49b9-6893-4d89-8271-6820379a7d04 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.689078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:14.720719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c2573-ecd4-4e23-8679-1244f863bd0f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.728994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.729648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.729926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.730215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] PATH_INFO: `/v3/users/3b3016d2e5a54c2ab25c45ed04034df1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.731388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b3016d2e5a54c2ab25c45ed04034df1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.733370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.738981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.740535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.740535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.740936 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.741442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:14.742101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.743833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.757159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1224/2454] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:14.758284 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.760135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.775600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac4865-6c9a-4638-ac2a-8327fab44432 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.782963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.784419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.784866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.785146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.785929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.787957 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.792707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b3016d2e5a54c2ab25c45ed04034df1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.801287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b3016d2e5a54c2ab25c45ed04034df1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.810413 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1225/2455] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/users => generated 292 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:14.811583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.822922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b3016d2e5a54c2ab25c45ed04034df1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.823748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b3016d2e5a54c2ab25c45ed04034df1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.824932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aee807-b193-47ec-9061-0d88beae8769 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.832498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b3016d2e5a54c2ab25c45ed04034df1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:14.833615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.834235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.834508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.834809 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.835452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.837364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:14.839268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1231/2456] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:14 2022] DELETE /identity/v3/users/3b3016d2e5a54c2ab25c45ed04034df1 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:14.842461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.842461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.874751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1226/2457] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:14.875735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:14.878213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:14.934003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fda224-1ee7-4af1-85c4-d17c5dee1e7d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.961614 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f9819a-4343-4d90-a755-5bf6cc6509a2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:14.968310 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:14.968927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:14.969390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:14.969680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:14.970302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:14.972456 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.010767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1227/2458] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:14 2022] POST /identity/v3/users => generated 286 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.012292 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.023807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.026807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.028141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.028427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.028692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.029353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.031031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.036916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.040703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c915f58-849d-4c34-b6a5-b2187a527130 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.052785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1232/2459] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:14 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:15.058657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.060901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.061776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.062229 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.062747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.063784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.066920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.073626 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76c49b1-ff60-4a34-9593-783d88250e49 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.085227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.085799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.086665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.086665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.087220 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.092997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.093237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1228/2460] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:15.093237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.094831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.121010 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1233/2461] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:15.121384 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8fe672-e709-4cab-a162-7cdfd3c761bd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.121615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.131952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.133170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.133883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.134191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] PATH_INFO: `/v3/projects/7f675af5f4354320a905f217fdc3c54a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.136305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f675af5f4354320a905f217fdc3c54a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.138926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89ee9b-bf67-44d1-800d-9e311e348670 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.140457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.157505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.158757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.159572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.160028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.161231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.165404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.186733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1234/2462] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:15.193644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.209402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c066f2c-b750-4fe1-9ca9-d451ea6aaf16 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.225914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f675af5f4354320a905f217fdc3c54a', 'request_id': 'req-3de63b6c-a5b5-46c0-9263-d119a74746ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.239155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f675af5f4354320a905f217fdc3c54a', 'request_id': 'req-3de63b6c-a5b5-46c0-9263-d119a74746ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.253209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f675af5f4354320a905f217fdc3c54a', 'request_id': 'req-3de63b6c-a5b5-46c0-9263-d119a74746ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.273243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] The token cache is being invalidate because project 7f675af5f4354320a905f217fdc3c54a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:15.273647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.274386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:15.276774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1229/2463] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:15 2022] DELETE /identity/v3/projects/7f675af5f4354320a905f217fdc3c54a => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:15.277604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.279182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.297129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.297787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.298061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.298353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] PATH_INFO: `/v3/projects/533127700c2e4aa3945c8197e678ff52/users/c34893f43daf42b1a0939691e394470d/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.315535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=533127700c2e4aa3945c8197e678ff52, user_id=c34893f43daf42b1a0939691e394470d, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.318241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f5ab05-4425-4fb1-a8bf-8343ce2c78e2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.334576 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1235/2464] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:15 2022] PUT /identity/v3/projects/533127700c2e4aa3945c8197e678ff52/users/c34893f43daf42b1a0939691e394470d/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:15.336418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c203bcea-08d3-4b87-b47f-cad2ec11cef4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.336711 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c203bcea-08d3-4b87-b47f-cad2ec11cef4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.336976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c203bcea-08d3-4b87-b47f-cad2ec11cef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.365529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c203bcea-08d3-4b87-b47f-cad2ec11cef4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:15.367115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c203bcea-08d3-4b87-b47f-cad2ec11cef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.369634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de63b6c-a5b5-46c0-9263-d119a74746ae admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.447232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1236/2465] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:15.452683 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.453676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.453955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.454233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] PATH_INFO: `/v3/projects/53de45de91a645adb4f14282f94fedc6/users/ac13970042a0460580fd3aa27cf35e5d/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.454428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19212442-9c37-4cda-86fb-d41fd02d1494 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.454628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19212442-9c37-4cda-86fb-d41fd02d1494 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.454947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19212442-9c37-4cda-86fb-d41fd02d1494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.474054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] RBAC: Authorizing `identity:create_grant(project_id=53de45de91a645adb4f14282f94fedc6, user_id=ac13970042a0460580fd3aa27cf35e5d, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.477096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d31a341-b6ad-4c62-8ff6-895ee1caf57d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.490972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-19212442-9c37-4cda-86fb-d41fd02d1494 None None] MFA Rules not processed for user `c34893f43daf42b1a0939691e394470d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:15.492617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-19212442-9c37-4cda-86fb-d41fd02d1494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.492982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1230/2466] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:15 2022] PUT /identity/v3/projects/53de45de91a645adb4f14282f94fedc6/users/ac13970042a0460580fd3aa27cf35e5d/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:15.494241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.494523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.494822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.514023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.520853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:15.522183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.555451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.588381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1237/2467] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:15.589788 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.590136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.590476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.603865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1231/2468] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:15.604902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.606322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.620013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9f57e-eb17-443e-afb9-2583fcbeaaa4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.624809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] MFA Rules not processed for user `ac13970042a0460580fd3aa27cf35e5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:15.624997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.625620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.625888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.626030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:15.626151 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.626854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.629590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.664521 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1232/2469] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/projects => generated 387 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.670612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.708037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1238/2470] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:15.709284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.720608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df59ed5-ee6e-4622-9d4b-90c20bb8fb56 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.743032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82344911-b144-456a-9bfb-f31e4705d725 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.747844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.748462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.748736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.749007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.749949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:15.750258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.751921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.764802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1233/2471] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:15.766145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.776476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d8ccb5-c139-4792-ad51-aed32f5e6faa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.781941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.782566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.782848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.783113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.783901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.785802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.787145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.787762 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.788039 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.788308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.788939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.790903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.816310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1239/2472] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/users => generated 300 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.817209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.817642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1234/2473] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.820460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.829811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2bc840-b2cb-481e-aa92-e3434f7ca4e5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.884930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.888474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363440f4-b4c4-4b3d-a896-fcaff2a9604f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.893486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.894216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.894458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.894603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.894797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.895133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.895426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.895536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.895705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.896347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.897383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.898092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.909672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1235/2474] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:15.910560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.915784 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:15.921945 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa50d42-36cc-4b40-8630-d3a8d2479010 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:15.928990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:15.929884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:15.930344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:15.930821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:15.931656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:15.931860 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1240/2475] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/projects => generated 365 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.933919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:15.936361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.961696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1236/2476] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/users => generated 293 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:15.961696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:15.970820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7abcc3-41b0-4972-adf0-b086d3c2415c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.010746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5787234a-2fdb-4ab9-bd5f-54bfc41524ce admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.017383 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.018631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.019452 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.019747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] PATH_INFO: `/v3/projects/4055151d88454fe0889a45450039bd7d/users/c30583cdae1c4c0a9b89b3461a4c4ac4/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.036897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=4055151d88454fe0889a45450039bd7d, user_id=c30583cdae1c4c0a9b89b3461a4c4ac4, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.038814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.039433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.039703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.039893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.039996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.040588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.042506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.053679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1241/2477] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:15 2022] PUT /identity/v3/projects/4055151d88454fe0889a45450039bd7d/users/c30583cdae1c4c0a9b89b3461a4c4ac4/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:16.054225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.067435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1237/2478] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:15 2022] POST /identity/v3/users => generated 283 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:16.071258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.081523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cd6feb-6337-45ea-ae87-d26e4ac30646 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.119986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a307808-100d-4ac9-bc45-8ca091cf33cc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.125279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.125881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.126243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.126480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.127941 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.129533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.139818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.140084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1242/2479] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:16.140669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.141072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.141447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.142207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.144643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.145219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.156729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1238/2480] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:16.157557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.157670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0179030c-4c33-4884-a31d-f2ba3a2b4b3b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.163367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.164307 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.164590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.164861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.165503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.167295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.169678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc65618-dd20-494f-8656-891fa1e4423f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.174571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.175247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.175524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.175799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] PATH_INFO: `/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06/users/39bf828f240c486bba853e6e8df4a29f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.178058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1243/2481] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:16.178058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.189616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3744978c-3fea-4a8d-a9cc-f95d33d775d0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.192508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33d3d6fd66d4f318b0fa8ea1152ed06, user_id=39bf828f240c486bba853e6e8df4a29f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.195230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.195385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.195897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.196198 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.196610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] PATH_INFO: `/v3/projects/4055151d88454fe0889a45450039bd7d/users/c30583cdae1c4c0a9b89b3461a4c4ac4/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.198911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4055151d88454fe0889a45450039bd7d, user_id=c30583cdae1c4c0a9b89b3461a4c4ac4, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.201749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.214474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1244/2482] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:16 2022] PUT /identity/v3/projects/4055151d88454fe0889a45450039bd7d/users/c30583cdae1c4c0a9b89b3461a4c4ac4/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:16.215297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.215657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1239/2483] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:16 2022] PUT /identity/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06/users/39bf828f240c486bba853e6e8df4a29f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:16.219451 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.270345 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:16.281441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:16.294827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13eca7d-c662-486b-ac90-5e3cd605fad6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.295696 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef5edde-892f-4cf0-be37-37c6002d56e7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.303587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.303587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.303587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.303587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.304070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.304165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4963381-1e29-4154-bc08-41c91721b200 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.304982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.305611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.305856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.306119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.306863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.308446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.315569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1245/2484] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:16.316972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db4c06-07af-4856-bee7-810e992860f3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.317250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db4c06-07af-4856-bee7-810e992860f3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.317527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db4c06-07af-4856-bee7-810e992860f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.320061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1240/2485] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:16.320956 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.331530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e1a21-6f2b-4b5c-ab97-3a6f462b2ab0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.336872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.337681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.338044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.338420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] PATH_INFO: `/v3/projects/aebd23a226e44b09abfba4ea35808192/users/7bb9375a21dc46e2b45b84214a2a269a/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.350850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-82db4c06-07af-4856-bee7-810e992860f3 None None] MFA Rules not processed for user `c30583cdae1c4c0a9b89b3461a4c4ac4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:16.352095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-82db4c06-07af-4856-bee7-810e992860f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.355361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aebd23a226e44b09abfba4ea35808192, user_id=7bb9375a21dc46e2b45b84214a2a269a, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.358045 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.372492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1241/2486] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:16 2022] PUT /identity/v3/projects/aebd23a226e44b09abfba4ea35808192/users/7bb9375a21dc46e2b45b84214a2a269a/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:16.373293 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.434378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1246/2487] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:16.435891 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adc17f-654a-4f87-a867-03bb34589ccd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.436181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adc17f-654a-4f87-a867-03bb34589ccd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.436446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35adc17f-654a-4f87-a867-03bb34589ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.449112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e943f1-6cd6-4ab7-b027-2cd7692c06a5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.453918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.454590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.454937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.455138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] PATH_INFO: `/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06/users/39bf828f240c486bba853e6e8df4a29f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.457662 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33d3d6fd66d4f318b0fa8ea1152ed06, user_id=39bf828f240c486bba853e6e8df4a29f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.460375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.466438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-35adc17f-654a-4f87-a867-03bb34589ccd None None] MFA Rules not processed for user `7bb9375a21dc46e2b45b84214a2a269a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:16.467723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-35adc17f-654a-4f87-a867-03bb34589ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.471834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1242/2488] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:16 2022] PUT /identity/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06/users/39bf828f240c486bba853e6e8df4a29f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:16.472642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.474218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.532757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1247/2489] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:16.534979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.535107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb988f-07d5-46d2-86ca-5c39b273a182 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.535258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.535523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.540287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.540925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.541205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.541476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.564073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] MFA Rules not processed for user `39bf828f240c486bba853e6e8df4a29f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:16.565293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.602699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:16.608146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.609867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.615699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:16.621126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1243/2490] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:16.622020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.638239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1248/2491] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:16.652047 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.653544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:16.668408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a2595d-6a46-4d71-8128-ca07c995c574 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.672222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.672831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.673111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.673381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.683621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.685059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.695515 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306dcc3f-0990-4298-b1c9-3b4099a597b2 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.697970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1249/2492] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:16.700422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.701265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.701398 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.701619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.702281 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.704193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.749812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1244/2493] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/projects => generated 365 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:16.754649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.766447 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dde6a9-b38c-47c5-8ec1-ddef4a6b74f3 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.864881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.865589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.865940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.866299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.867427 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.871331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.904943 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1250/2494] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:16 2022] POST /identity/v3/users => generated 289 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:16.906179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.927235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53ed7d6-6727-4aec-906f-e1d9ebd29b04 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.933208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.934109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.934587 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.934967 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:16.935777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:16.937468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:16.964209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1245/2495] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:16 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:16.973110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:16.984958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5864aff-e622-4f5b-95ab-8b3cafa302f1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:16.991958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:16.992714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:16.993338 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:16.993656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] PATH_INFO: `/v3/projects/fd79204541a74073afd02f5fc7ae60d0/users/30024b364be54f78949a6761535b8f0f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.019996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd79204541a74073afd02f5fc7ae60d0, user_id=30024b364be54f78949a6761535b8f0f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.024407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.040859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1251/2496] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:16 2022] PUT /identity/v3/projects/fd79204541a74073afd02f5fc7ae60d0/users/30024b364be54f78949a6761535b8f0f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:17.044879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.058411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c3527-c8c0-4a8d-9767-845ba60e2183 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.141295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.142067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.142441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.142980 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.144140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.146070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.165547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1246/2497] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:17.168108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.180556 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890dd6ab-6d0e-4ec8-9cca-14c2f61f9957 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.185475 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.186377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.186818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.187129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] PATH_INFO: `/v3/projects/fd79204541a74073afd02f5fc7ae60d0/users/30024b364be54f78949a6761535b8f0f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.189682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd79204541a74073afd02f5fc7ae60d0, user_id=30024b364be54f78949a6761535b8f0f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.193134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1367cef6-9b35-4d5f-8e76-91388112970f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.204698 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.208856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1252/2498] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:17 2022] PUT /identity/v3/projects/fd79204541a74073afd02f5fc7ae60d0/users/30024b364be54f78949a6761535b8f0f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:17.219481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386e148-991e-4d50-b0db-4a8e85330dc6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.220984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.221293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.221904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.258856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] MFA Rules not processed for user `30024b364be54f78949a6761535b8f0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:17.260509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.295533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.296104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.297134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.297134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] PATH_INFO: `/v3/users/ac13970042a0460580fd3aa27cf35e5d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.297910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac13970042a0460580fd3aa27cf35e5d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.307915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.314466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:17.349508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1253/2499] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:17.356704 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.367728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac13970042a0460580fd3aa27cf35e5d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.367937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8628728e-74d9-4dc0-a97c-e83e8bd23c44 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.372747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.373407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.373858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.374147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] PATH_INFO: `/v3/users/c34893f43daf42b1a0939691e394470d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.374962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac13970042a0460580fd3aa27cf35e5d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.375346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=c34893f43daf42b1a0939691e394470d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.378284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.383512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:17.388566 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac13970042a0460580fd3aa27cf35e5d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.389378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac13970042a0460580fd3aa27cf35e5d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.390060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac13970042a0460580fd3aa27cf35e5d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.404327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1247/2500] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:17 2022] DELETE /identity/v3/users/ac13970042a0460580fd3aa27cf35e5d => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:17.405160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.406743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.447117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c34893f43daf42b1a0939691e394470d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.454208 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c34893f43daf42b1a0939691e394470d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.467770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c34893f43daf42b1a0939691e394470d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.468591 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c34893f43daf42b1a0939691e394470d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.476881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ccb3f020-4f5d-4bc6-932e-453f9da764ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c34893f43daf42b1a0939691e394470d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:17.483772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1254/2501] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:17 2022] DELETE /identity/v3/users/c34893f43daf42b1a0939691e394470d => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:17.485041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e3342-c604-4d93-a524-1142e0b27e2a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.485312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e3342-c604-4d93-a524-1142e0b27e2a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.485449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6076e2ac-21a2-4cbf-ad05-1617d999c95e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.485577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e3342-c604-4d93-a524-1142e0b27e2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.505805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1c8e3342-c604-4d93-a524-1142e0b27e2a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:17.506963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1c8e3342-c604-4d93-a524-1142e0b27e2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.540004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.540712 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.540898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.541168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.543453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.575356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1255/2502] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:17.576792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.577083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.577358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.598735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:17.599957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.602036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:17.607885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1256/2503] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:17.608836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.610059 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.621226 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.622714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.633498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1248/2504] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:17.634275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.635858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.678682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d1de5f-5146-4421-a28b-6cc59ce0fb6e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.682587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.683202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.683485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.683760 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.694526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.696031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.704371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45714fb6-62cf-4dd2-9a8b-5e3405f0a816 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.707256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1257/2505] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:17.714224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.714877 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.715152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.715879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.718239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:17.719898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.726268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.727888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.735757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64d88e6-bdc8-45fc-b222-3f2723f9f839 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:17.740636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1249/2506] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:17.741150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:17.741912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:17.742213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:17.742467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:17.744461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:17.763975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:17.828922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:17.831777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:17.845546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1258/2507] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:17 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:18.065028 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.066865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.081520 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.082079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5385be0f-1af0-4721-ab06-6ab9858a31f3 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.085753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.086949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.087643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.087927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.088201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] PATH_INFO: `/v3/projects/53de45de91a645adb4f14282f94fedc6` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.089530 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53de45de91a645adb4f14282f94fedc6)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.091640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.104602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f02028-0ff3-4f86-b2e2-2d6f34ac0f66 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.109260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.109960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.110267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.110585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] PATH_INFO: `/v3/projects/533127700c2e4aa3945c8197e678ff52` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.111824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=533127700c2e4aa3945c8197e678ff52)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.115151 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.167470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53de45de91a645adb4f14282f94fedc6', 'request_id': 'req-1bfb5fd3-43f3-4378-948b-6196507785b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.173357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '533127700c2e4aa3945c8197e678ff52', 'request_id': 'req-e0467337-3397-45e1-82af-a89732f296f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.173783 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53de45de91a645adb4f14282f94fedc6', 'request_id': 'req-1bfb5fd3-43f3-4378-948b-6196507785b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.181086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '533127700c2e4aa3945c8197e678ff52', 'request_id': 'req-e0467337-3397-45e1-82af-a89732f296f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.183021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53de45de91a645adb4f14282f94fedc6', 'request_id': 'req-1bfb5fd3-43f3-4378-948b-6196507785b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.188426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '533127700c2e4aa3945c8197e678ff52', 'request_id': 'req-e0467337-3397-45e1-82af-a89732f296f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.195257 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] The token cache is being invalidate because project 53de45de91a645adb4f14282f94fedc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:18.195546 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.196223 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.197744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1250/2508] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/projects/53de45de91a645adb4f14282f94fedc6 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:18.200733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] The token cache is being invalidate because project 533127700c2e4aa3945c8197e678ff52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:18.200733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.200733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.201913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.202250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1259/2509] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/projects/533127700c2e4aa3945c8197e678ff52 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:18.203387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.206496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.207833 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.270512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb5fd3-43f3-4378-948b-6196507785b1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.275352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.276016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.276290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.276558 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] PATH_INFO: `/v3/users/39bf828f240c486bba853e6e8df4a29f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.277692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39bf828f240c486bba853e6e8df4a29f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.277807 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0467337-3397-45e1-82af-a89732f296f2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.279810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.282293 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.283209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.284095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.284491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] PATH_INFO: `/v3/users/c30583cdae1c4c0a9b89b3461a4c4ac4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.285629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c30583cdae1c4c0a9b89b3461a4c4ac4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.287569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.298443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:18.334091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:18.334304 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39bf828f240c486bba853e6e8df4a29f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.337740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c30583cdae1c4c0a9b89b3461a4c4ac4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.341586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39bf828f240c486bba853e6e8df4a29f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.347492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c30583cdae1c4c0a9b89b3461a4c4ac4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.355692 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39bf828f240c486bba853e6e8df4a29f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.356436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39bf828f240c486bba853e6e8df4a29f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.357057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39bf828f240c486bba853e6e8df4a29f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.360133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c30583cdae1c4c0a9b89b3461a4c4ac4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.361002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c30583cdae1c4c0a9b89b3461a4c4ac4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.370516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c30583cdae1c4c0a9b89b3461a4c4ac4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.374116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1251/2510] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/users/39bf828f240c486bba853e6e8df4a29f => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:18.377530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1260/2511] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/users/c30583cdae1c4c0a9b89b3461a4c4ac4 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:18.761773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.763512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.769874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:18.773600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:18.871567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96d530b-904c-4835-a1d2-578780956cf0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.878248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f82ca4b-536f-4e40-9128-87ae7c3f1710 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:18.879126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.880110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.880562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.881015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] PATH_INFO: `/v3/projects/4055151d88454fe0889a45450039bd7d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.883207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4055151d88454fe0889a45450039bd7d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.885775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:18.886694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.886811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.887202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.887619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] PATH_INFO: `/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.889314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f33d3d6fd66d4f318b0fa8ea1152ed06)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:18.892557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:18.956667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4055151d88454fe0889a45450039bd7d', 'request_id': 'req-43fd86de-b3d5-4749-927f-d74cdcb299d8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.957074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f33d3d6fd66d4f318b0fa8ea1152ed06', 'request_id': 'req-47f447ac-f49f-4949-bf50-22e264e29f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.964316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f33d3d6fd66d4f318b0fa8ea1152ed06', 'request_id': 'req-47f447ac-f49f-4949-bf50-22e264e29f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.966236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4055151d88454fe0889a45450039bd7d', 'request_id': 'req-43fd86de-b3d5-4749-927f-d74cdcb299d8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.974670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f33d3d6fd66d4f318b0fa8ea1152ed06', 'request_id': 'req-47f447ac-f49f-4949-bf50-22e264e29f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.977455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4055151d88454fe0889a45450039bd7d', 'request_id': 'req-43fd86de-b3d5-4749-927f-d74cdcb299d8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.987296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] The token cache is being invalidate because project f33d3d6fd66d4f318b0fa8ea1152ed06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:18.987648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.988281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-47f447ac-f49f-4949-bf50-22e264e29f8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.989652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] The token cache is being invalidate because project 4055151d88454fe0889a45450039bd7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:18.989940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.990194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1261/2512] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/projects/f33d3d6fd66d4f318b0fa8ea1152ed06 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:18.990699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43fd86de-b3d5-4749-927f-d74cdcb299d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:18.992169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1252/2513] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:18 2022] DELETE /identity/v3/projects/4055151d88454fe0889a45450039bd7d => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:18.994948 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49016aab-0d5c-4dbd-bc9e-db7b0367fe8f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.995231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49016aab-0d5c-4dbd-bc9e-db7b0367fe8f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.995531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49016aab-0d5c-4dbd-bc9e-db7b0367fe8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:18.996678 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60534d68-0d43-4d74-aee7-a1f19de8f64e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:18.996960 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60534d68-0d43-4d74-aee7-a1f19de8f64e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:18.997231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60534d68-0d43-4d74-aee7-a1f19de8f64e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.015668 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-49016aab-0d5c-4dbd-bc9e-db7b0367fe8f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.016864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-60534d68-0d43-4d74-aee7-a1f19de8f64e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.016970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-49016aab-0d5c-4dbd-bc9e-db7b0367fe8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.018206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-60534d68-0d43-4d74-aee7-a1f19de8f64e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.040255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:19.073335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:19.092172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1262/2514] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:19.101455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.102681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.102955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.115462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1253/2515] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:19.123291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.123675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.124053 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.125204 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.126585 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.144970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1263/2516] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:19.148226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.154115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.155965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.163478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51f944-7f2c-4d17-8951-d03869d4cff2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.166607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1254/2517] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:19.170714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.171757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.172379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.172661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.172936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.173594 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:19.173889 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.176698 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.184063 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4142980-c332-45d2-b6d0-b023c5438844 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.189652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.190610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.190995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.191365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.191629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1264/2518] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:19.192252 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:19.192644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.194557 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.196399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.207089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4754be-7de7-45f0-be97-c283413f914d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.208165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1255/2519] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:19.212100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.212710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.212987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.213124 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.213245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.213872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.215867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.223948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36f6b18-dd8c-481c-8756-f3515a182042 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.228601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.229270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.229762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.230039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.230709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.232620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.258144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1265/2520] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/projects => generated 387 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:19.262371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.266694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1256/2521] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:19.269611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.337086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c5dc47-c934-418a-a770-f12435a01d81 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.342593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.342783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.343074 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.343382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.344008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.344580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441e59b-5629-422b-a291-5000cc3b02f1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.346642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.351171 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.352159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.352463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.352764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.353471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.355604 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.372855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1266/2522] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/users => generated 294 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:19.377710 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.385016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1257/2523] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/users => generated 290 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:19.389352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.389543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feda1c2-ad09-4841-8a13-414f1daeac4f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.400356 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.401014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.401273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2b71c-31c8-4e09-a322-fdfcefae90f2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.401538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.401814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.403284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.405410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.407462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.408395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.414942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.414942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.414942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.414942 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.426155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1267/2524] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:19.428061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1258/2525] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:19.430095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.431920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.442794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c48c52-908a-49ae-881c-8dbddbceb4db admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.444879 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba33d81-6f47-4ccc-81e2-c6ba036b8d1b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.448599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.449298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.449644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.449985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.450745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.453202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.454216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.455111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.456013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.456748 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.457113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.458827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.475062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1268/2526] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:19.476095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.477152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1259/2527] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:19.478162 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.486849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e999753b-a3e8-4024-abbb-f03ef7a953a4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.493037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9587470d-3c51-4ddb-9ea4-74233b88f018 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.495486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.496176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.496548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.496707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] PATH_INFO: `/v3/projects/ceeb8683cf0445dca652d70cd9782007/users/5b7c7805d26443f8ae443555d5e1dd2f/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.501444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.502631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.503259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.503800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] PATH_INFO: `/v3/projects/6ce9d2ad0eb94ce591cb65ae15c327da/users/630642706f2546f29e6dcb63a336ae0f/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.514384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ceeb8683cf0445dca652d70cd9782007, user_id=5b7c7805d26443f8ae443555d5e1dd2f, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.519021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21966cda-3665-44b5-b844-c73efdeeacf8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.525225 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ce9d2ad0eb94ce591cb65ae15c327da, user_id=630642706f2546f29e6dcb63a336ae0f, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.529208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4551a8-9f52-437d-9e47-1a7a202556ca admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.539715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1269/2528] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:19 2022] PUT /identity/v3/projects/ceeb8683cf0445dca652d70cd9782007/users/5b7c7805d26443f8ae443555d5e1dd2f/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:19.550038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.550605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.551149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.553118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1260/2529] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:19 2022] PUT /identity/v3/projects/6ce9d2ad0eb94ce591cb65ae15c327da/users/630642706f2546f29e6dcb63a336ae0f/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:19.563447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.563727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.564003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.587637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] MFA Rules not processed for user `5b7c7805d26443f8ae443555d5e1dd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.588856 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.592825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] MFA Rules not processed for user `630642706f2546f29e6dcb63a336ae0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:19.594009 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:19.609046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:19.635322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:19.662488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1270/2530] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:19.666790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.667135 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1261/2531] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:19.671227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.730738 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a688ec25-c3a2-43fc-8d53-ce3b5c9a3d4b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.735483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3360e2-9e27-40a8-9585-a5d4a2b075bc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.737884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.738574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.738855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.739122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.740023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.741561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.741916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.742167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.742569 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.742742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.743372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.745264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.775617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1271/2532] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:19.776176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1262/2533] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/projects => generated 379 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:19.777312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.779962 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.857023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128e0ae3-d3fa-4e18-a6da-ad5000946a4c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.926602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.927296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.927580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.927852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.928517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.930500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.959118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1272/2534] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/users => generated 293 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:19.960444 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:19.967094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:19.972243 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2228e8d8-48f8-4825-b702-9dfa55473686 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.978803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.979490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.979745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9518fef-514b-4339-8073-4a45ac4534a5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:19.979886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.980223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.980939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.982640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.984791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:19.985389 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:19.985657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:19.985920 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:19.987386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:19.989561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:19.994389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1273/2535] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:19.998274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.009481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bedb297-94b5-4c88-8a7e-3c09d265e6ae admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.013068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1263/2536] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:19 2022] POST /identity/v3/users => generated 296 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:20.016982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.017291 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.017767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.018203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.018715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.019458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.021051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.027640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f1758-9f1e-4769-8301-dc17c1a5c8c5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.032269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.032433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1274/2537] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:20.032859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.033129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.033444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.034072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.035700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.036577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.046012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd1887-d214-4ac5-bb0b-6c9ecfa5aa79 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.046217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1264/2538] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:20.051041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.052040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.052681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.052954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.053226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] PATH_INFO: `/v3/projects/63ea233e626042c7a2f01ff2f9777161/users/773543a5fe054e7a83af91582055ff0e/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.061640 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ee71c6-8f0f-4973-bda6-193b39c4e8f6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.066687 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.067331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.067605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.067875 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] PATH_INFO: `/v3/projects/76cfb4910a9f4bd083d71a391166c272/users/baeddd74847545039c5a63c9456d0cf8/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.068247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63ea233e626042c7a2f01ff2f9777161, user_id=773543a5fe054e7a83af91582055ff0e, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.070918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b129788c-85ed-48a9-8ef2-01d8a2d77068 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.084873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76cfb4910a9f4bd083d71a391166c272, user_id=baeddd74847545039c5a63c9456d0cf8, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.086917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1275/2539] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:20 2022] PUT /identity/v3/projects/63ea233e626042c7a2f01ff2f9777161/users/773543a5fe054e7a83af91582055ff0e/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:20.087733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.091097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:20.098940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.099225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.099497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.103998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1265/2540] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:20 2022] PUT /identity/v3/projects/76cfb4910a9f4bd083d71a391166c272/users/baeddd74847545039c5a63c9456d0cf8/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:20.109765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.131420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] MFA Rules not processed for user `773543a5fe054e7a83af91582055ff0e`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:20.134915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:20.176232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef6958-cccb-48b5-b4da-90ee6abefa74 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.180864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.181457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.181732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.181996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.182655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.184228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.194409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1266/2541] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:20.198724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.208679 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1276/2542] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:20.209457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ca77ae-6262-4841-b728-7ac6ec348c54 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.213037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.214590 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.215235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.215507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.216156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] PATH_INFO: `/v3/projects/76cfb4910a9f4bd083d71a391166c272/users/baeddd74847545039c5a63c9456d0cf8/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.218051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76cfb4910a9f4bd083d71a391166c272, user_id=baeddd74847545039c5a63c9456d0cf8, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.220689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065a7f2-053e-427b-87ad-85e3e0ea4b57 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.223879 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aca5618-5556-4a19-8131-d1333dd08812 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.228586 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.229185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.229462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.229732 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.230357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.232292 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.232495 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1267/2543] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:20 2022] PUT /identity/v3/projects/76cfb4910a9f4bd083d71a391166c272/users/baeddd74847545039c5a63c9456d0cf8/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:20.244536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.244807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.245073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.264207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1277/2544] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:20 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:20.265277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.275446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] MFA Rules not processed for user `baeddd74847545039c5a63c9456d0cf8`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:20.277078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:20.335159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf857b77-d425-4b8d-90f1-2e3a4b788915 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.340167 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.340785 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.341051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.341334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.342086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.344044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.351833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1268/2545] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:20 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:20.364645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.366058 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:20.370584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1278/2546] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:20 2022] POST /identity/v3/users => generated 300 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:20.372645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.383413 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b142c1c-acdb-49d9-975f-a67a0cfaf23c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.388054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.388660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.388937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.389230 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.389982 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.391590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.406181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1279/2547] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:20.408472 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.414830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:20.419630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b6ee80-792d-429f-8390-f3e4e8f9669c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.428845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.429567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.429843 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.430109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] PATH_INFO: `/v3/projects/fd6650884b42467795f08b4f3f849457/users/13869e3d6e374fd8812b5e44bc0ac3f2/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.438878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798d917-3853-4288-a189-4fc86dcaa833 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.443368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.443983 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.444253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.444523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.447273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd6650884b42467795f08b4f3f849457, user_id=13869e3d6e374fd8812b5e44bc0ac3f2, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.450113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.454981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.457457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.466265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1280/2548] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:20 2022] PUT /identity/v3/projects/fd6650884b42467795f08b4f3f849457/users/13869e3d6e374fd8812b5e44bc0ac3f2/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:20.469294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1269/2549] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:20.470767 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.546396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a91bf25-b53b-4158-a036-2ee0a67d7dae admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.552092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.552730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.553011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.553322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.553950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.555962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfb4be-dec4-4516-ae3b-b25d3e084aea admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.570720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1281/2550] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:20.575468 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.590519 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab59a3d-94bd-4195-bca0-267a23acbe11 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.595786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.598596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.598596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.598596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] PATH_INFO: `/v3/projects/fd6650884b42467795f08b4f3f849457/users/13869e3d6e374fd8812b5e44bc0ac3f2/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.601436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd6650884b42467795f08b4f3f849457, user_id=13869e3d6e374fd8812b5e44bc0ac3f2, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.604804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.613285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:20.645068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1270/2551] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:20 2022] PUT /identity/v3/projects/fd6650884b42467795f08b4f3f849457/users/13869e3d6e374fd8812b5e44bc0ac3f2/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:20.655976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.656173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.656449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.689042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] MFA Rules not processed for user `13869e3d6e374fd8812b5e44bc0ac3f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:20.691251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:20.793674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1282/2552] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:20 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:20.809666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:20.874778 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce8032c-3a80-4373-95dd-1acfe8749656 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:20.879146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:20.879761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:20.880047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:20.880315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:20.889559 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:20.891020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bd77be-561b-4b0e-b23a-f538518fae80 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:20.901271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1271/2553] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:20 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:21.110095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.201589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6d1722-b2c7-409e-9db8-9dc3dc4e94ce None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.207471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.208144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.208435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.208855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] PATH_INFO: `/v3/users/630642706f2546f29e6dcb63a336ae0f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.210193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=630642706f2546f29e6dcb63a336ae0f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.211977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.216298 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:21.296864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '630642706f2546f29e6dcb63a336ae0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.306954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '630642706f2546f29e6dcb63a336ae0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.321615 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '630642706f2546f29e6dcb63a336ae0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.323064 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '630642706f2546f29e6dcb63a336ae0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.334571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '630642706f2546f29e6dcb63a336ae0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.344160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1283/2554] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:21 2022] DELETE /identity/v3/users/630642706f2546f29e6dcb63a336ae0f => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:21.350048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.350564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.351038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.373626 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:21.374870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.438019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1272/2555] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:21.448505 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.449962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.515143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c80d5d-68aa-49a0-b708-9e862f1b5aa5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.520303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.521014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.521741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.521741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.531202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.532763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.543357 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1284/2556] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:21.583647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.585007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.654830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0ff83-223d-450c-a2d6-bfc664528cf0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.659131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.660129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.660442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.660719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.662516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.717270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:21.730260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.731727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.744681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1273/2557] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:21 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:21.844832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:21.847004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:21.871510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f85c19-983b-4087-b648-03b310b56b07 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:21.877958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:21.878716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:21.879066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:21.879410 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] PATH_INFO: `/v3/projects/6ce9d2ad0eb94ce591cb65ae15c327da` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:21.889426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ce9d2ad0eb94ce591cb65ae15c327da)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:21.891459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:21.957469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ce9d2ad0eb94ce591cb65ae15c327da', 'request_id': 'req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.965927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ce9d2ad0eb94ce591cb65ae15c327da', 'request_id': 'req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.976685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ce9d2ad0eb94ce591cb65ae15c327da', 'request_id': 'req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.989041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] The token cache is being invalidate because project 6ce9d2ad0eb94ce591cb65ae15c327da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:21.989598 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.990343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:21.992396 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1285/2558] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:21 2022] DELETE /identity/v3/projects/6ce9d2ad0eb94ce591cb65ae15c327da => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:21.996870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:22.008553 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a208065-b7ad-43a0-9dd1-588f7a1a60c8 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:22.076341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:22.077129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.077641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.077998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] PATH_INFO: `/v3/users/baeddd74847545039c5a63c9456d0cf8` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.079491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baeddd74847545039c5a63c9456d0cf8)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:22.082110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:22.136168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baeddd74847545039c5a63c9456d0cf8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.144400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baeddd74847545039c5a63c9456d0cf8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.165730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baeddd74847545039c5a63c9456d0cf8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.166510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baeddd74847545039c5a63c9456d0cf8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.167150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f8e1894-f0a0-42e5-868b-b121f82e9234 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baeddd74847545039c5a63c9456d0cf8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.180617 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1274/2559] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:21 2022] DELETE /identity/v3/users/baeddd74847545039c5a63c9456d0cf8 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:22.495808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:22.497492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:22.511546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17544b9d-493c-4f91-9d7e-871b4f8c8c8e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:22.515351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:22.575316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:22.575951 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.576226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.576495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] PATH_INFO: `/v3/projects/76cfb4910a9f4bd083d71a391166c272` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.577611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76cfb4910a9f4bd083d71a391166c272)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:22.579641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:22.630669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76cfb4910a9f4bd083d71a391166c272', 'request_id': 'req-efb31ef0-37fd-4614-8e0f-5927390790e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.637108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76cfb4910a9f4bd083d71a391166c272', 'request_id': 'req-efb31ef0-37fd-4614-8e0f-5927390790e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.645700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76cfb4910a9f4bd083d71a391166c272', 'request_id': 'req-efb31ef0-37fd-4614-8e0f-5927390790e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.660279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] The token cache is being invalidate because project 76cfb4910a9f4bd083d71a391166c272 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:22.660722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.661815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efb31ef0-37fd-4614-8e0f-5927390790e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:22.663908 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1286/2560] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:22 2022] DELETE /identity/v3/projects/76cfb4910a9f4bd083d71a391166c272 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:22.669087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.669379 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.669648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.691574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:22.692784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:22.758079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1275/2561] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:22.766045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.766330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.766671 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.791069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:22.792541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:22.805488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1287/2562] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:22.810346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:22.820444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777c69a-0d29-4b70-a314-cbb177ab9f3a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:22.825044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:22.825952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.826233 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.826501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.827175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:22.827470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:22.829059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:22.836288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:22.848603 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1276/2563] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:22.850015 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:22.859405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabeeb05-0c2e-4a43-b579-4684b7d482ad None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:22.865351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:22.865949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.866264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.866533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.867275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:22.869163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:22.897461 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1288/2564] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:22 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:22.902261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:22.913021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10ad92-1ab6-447b-a748-05aed1c09f62 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:22.982407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:22.983056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:22.983334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:22.983611 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:22.984249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:22.986179 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.009948 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1277/2565] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:22 2022] POST /identity/v3/users => generated 291 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:23.014086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.025341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b06dbc9-7264-485c-aa2b-1f5ed37fd984 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.030013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.030645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.030924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.031194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.031840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.034145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.045527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1289/2566] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:23.050186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.060442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db2708-b4fb-4b7a-8ff2-fcda55da9e5a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.065598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.066215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.066681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.066828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.067500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.069057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aeef44-809a-48ed-9d1e-ae119f28cf03 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.080367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1278/2567] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:23.084823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.095883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ec921-06ad-460f-8573-96276f66fd14 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.101076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.102173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.102173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.102448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] PATH_INFO: `/v3/projects/412bd735ddf64839bfa8d8f3e0c8191d/users/4bbf27f7ce13480c87ee3d9c348112a1/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.119441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=412bd735ddf64839bfa8d8f3e0c8191d, user_id=4bbf27f7ce13480c87ee3d9c348112a1, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.122231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.139606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1290/2568] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:23 2022] PUT /identity/v3/projects/412bd735ddf64839bfa8d8f3e0c8191d/users/4bbf27f7ce13480c87ee3d9c348112a1/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:23.151238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.151564 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.151802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.182944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] MFA Rules not processed for user `4bbf27f7ce13480c87ee3d9c348112a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:23.184185 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:23.256373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1279/2569] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:23 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:23.261046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.296022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:23.326852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f960de-6f89-4476-bd3b-cb0fc61adc14 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.331623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.332229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.332526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.332786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.333425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.335572 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.366853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1291/2570] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:23 2022] POST /identity/v3/projects => generated 383 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:23.371024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.382112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775733f-2fc0-4120-ace8-32b8ec25b1ea None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.439348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.440254 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.440654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.440933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.441584 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.443770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.456770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:23.478584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.479207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1280/2571] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:23 2022] POST /identity/v3/users => generated 298 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:23.489846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93089f95-8f25-4fb6-a819-b32b04a9e5d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.494959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.495733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.496014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.496677 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.497319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.499346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.510076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1292/2572] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:23.514628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.524567 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e854206-b8a2-43e6-bdf0-3fdc4d4c2751 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.529182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.529821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.530188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.530421 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] PATH_INFO: `/v3/projects/a0380f7ed3fb4feaa416080d944bcb15/users/7732bb92b6094811a0f2b471398fd9f6/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.546420 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0380f7ed3fb4feaa416080d944bcb15, user_id=7732bb92b6094811a0f2b471398fd9f6, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.550127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.562565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1281/2573] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:23 2022] PUT /identity/v3/projects/a0380f7ed3fb4feaa416080d944bcb15/users/7732bb92b6094811a0f2b471398fd9f6/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:23.566911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.576983 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3975af-8ae2-4e13-9647-8a932f4cb68d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.636646 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.637417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.637580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.637845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.638512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.640178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf90d16c-3799-40f2-a0ab-cf5938f30f8a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.653369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1293/2574] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:23.657910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.669533 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752c59e8-614a-497a-b29a-436c559b3944 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.675395 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.676090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.676385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.676651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] PATH_INFO: `/v3/projects/a0380f7ed3fb4feaa416080d944bcb15/users/7732bb92b6094811a0f2b471398fd9f6/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.678646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0380f7ed3fb4feaa416080d944bcb15, user_id=7732bb92b6094811a0f2b471398fd9f6, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.681246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.691874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1282/2575] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:23 2022] PUT /identity/v3/projects/a0380f7ed3fb4feaa416080d944bcb15/users/7732bb92b6094811a0f2b471398fd9f6/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:23.759532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.759819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.760246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.793234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] MFA Rules not processed for user `7732bb92b6094811a0f2b471398fd9f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:23.794362 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:23.873852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1294/2576] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:23 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:23.890246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:23.892076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:23.953214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dc4e2-0bfc-4b82-a26c-ccdd87cf1d6b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:23.957825 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:23.958423 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:23.958725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:23.958992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:23.969278 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:23.970840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:23.982308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1283/2577] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:23 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:24.054858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:24.064451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e3812-2d2c-49ba-851e-73fe38571bc9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:24.068277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:24.068864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:24.069135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:24.069399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:24.095593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:24.132618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:24.132919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfce106-40f4-47cc-a07f-bc7d387ca2e0 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:24.143404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1295/2578] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:24 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:24.452155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:24.516049 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f139c6e-f480-4caa-b159-1cb0a57d932c service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:24.520504 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:24.521120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:24.521388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:24.521653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] PATH_INFO: `/v3/users/4bbf27f7ce13480c87ee3d9c348112a1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:24.523005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4bbf27f7ce13480c87ee3d9c348112a1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:24.525050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:24.574324 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4bbf27f7ce13480c87ee3d9c348112a1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:24.584113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4bbf27f7ce13480c87ee3d9c348112a1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:24.589660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:24.599348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4bbf27f7ce13480c87ee3d9c348112a1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:24.601704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4bbf27f7ce13480c87ee3d9c348112a1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:24.602346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4bbf27f7ce13480c87ee3d9c348112a1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:24.617426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1284/2579] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:24 2022] DELETE /identity/v3/users/4bbf27f7ce13480c87ee3d9c348112a1 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:24.630193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:24.630193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:24.630498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:24.652162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:24.653437 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:24.718900 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1296/2580] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:24.729906 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:24.731585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:24.794985 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae482233-1d48-4288-957f-e88b1a03519d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:24.799750 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:24.800309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:24.800582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:24.800848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:24.810196 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:24.811721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:24.822651 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1285/2581] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:25.153724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:25.155648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:25.170498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b72038-79cd-4b69-9f1e-0c2c6b7c0ffb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:25.178420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:25.178700 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:25.179031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:25.179239 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] PATH_INFO: `/v3/projects/412bd735ddf64839bfa8d8f3e0c8191d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:25.180423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=412bd735ddf64839bfa8d8f3e0c8191d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:25.182454 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:25.242965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '412bd735ddf64839bfa8d8f3e0c8191d', 'request_id': 'req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.250009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '412bd735ddf64839bfa8d8f3e0c8191d', 'request_id': 'req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.257492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '412bd735ddf64839bfa8d8f3e0c8191d', 'request_id': 'req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.269552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] The token cache is being invalidate because project 412bd735ddf64839bfa8d8f3e0c8191d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:25.269855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.270509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.272026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1297/2582] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:25 2022] DELETE /identity/v3/projects/412bd735ddf64839bfa8d8f3e0c8191d => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:25.276081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:25.287057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3963bce-a2fc-4614-bb16-b3b5b6d11f4c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:25.352404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:25.353091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:25.353432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:25.353759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] PATH_INFO: `/v3/users/7732bb92b6094811a0f2b471398fd9f6` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:25.355129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7732bb92b6094811a0f2b471398fd9f6)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:25.357036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:25.409876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7732bb92b6094811a0f2b471398fd9f6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.417537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7732bb92b6094811a0f2b471398fd9f6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.431299 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7732bb92b6094811a0f2b471398fd9f6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.432136 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7732bb92b6094811a0f2b471398fd9f6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.432814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-793ad235-1401-475b-a2cb-e21a1a268617 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7732bb92b6094811a0f2b471398fd9f6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.447764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1286/2583] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:25 2022] DELETE /identity/v3/users/7732bb92b6094811a0f2b471398fd9f6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:25.740752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:25.742502 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:25.757508 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3347807c-e461-4c3c-a0ac-e7a8ab3529d5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:25.812648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:25.824471 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:25.825128 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:25.825399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:25.825669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] PATH_INFO: `/v3/projects/a0380f7ed3fb4feaa416080d944bcb15` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:25.826814 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0380f7ed3fb4feaa416080d944bcb15)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:25.828709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:25.875029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0380f7ed3fb4feaa416080d944bcb15', 'request_id': 'req-74a0217a-136f-407e-8ba8-131245648160'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.881347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0380f7ed3fb4feaa416080d944bcb15', 'request_id': 'req-74a0217a-136f-407e-8ba8-131245648160'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.889854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0380f7ed3fb4feaa416080d944bcb15', 'request_id': 'req-74a0217a-136f-407e-8ba8-131245648160'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.902300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] The token cache is being invalidate because project a0380f7ed3fb4feaa416080d944bcb15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:25.902621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.903280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-74a0217a-136f-407e-8ba8-131245648160 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:25.904791 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1298/2584] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:25 2022] DELETE /identity/v3/projects/a0380f7ed3fb4feaa416080d944bcb15 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:25.910238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:25.910731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:25.910846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:25.918605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:25.933713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:25.935168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:26.001023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1287/2585] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:26.009187 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.009466 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.009880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.042759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:26.043994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:26.052491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1299/2586] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:26.056822 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.066779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0399d-d467-4700-b13a-b7034a457709 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.071330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.071924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.072199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.072467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.073097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:26.073400 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.075021 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.087751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1288/2587] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:26.093546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.104544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e40ae02-b6f1-400b-9579-2bba87d1bba2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.109227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.109836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.110110 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.110376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.111044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.112986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.155692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1300/2588] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:26 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:26.156751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.167541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018803d-cb01-42cc-9316-94836e01ce22 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.225424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.226033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.226308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.226595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.227315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.229596 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.255032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1289/2589] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:26 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:26.259349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.271606 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf2e4c6-b4ad-4b14-b1ad-56da22ef1f2d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.276724 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.277390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.277675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.277962 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.278682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.280497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.296314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1301/2590] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:26 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:26.300759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.323717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21be2521-1db2-4117-b846-db185ca34c8d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.323717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.323717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.323717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.324122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.324122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.324122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a2d413-2b8c-4b0b-9f69-18516831c8d1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.333715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1290/2591] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:26 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:26.338478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.351681 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af807a3f-edd1-470e-9fa5-695304159cc3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.357425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.358153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.358571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.358853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] PATH_INFO: `/v3/projects/bdf5f8fc46c945348c88540f6217bd64/users/250596c2774b4e46a76c9dcb2b619fc6/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.375597 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf5f8fc46c945348c88540f6217bd64, user_id=250596c2774b4e46a76c9dcb2b619fc6, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.378829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.392179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1302/2592] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:26 2022] PUT /identity/v3/projects/bdf5f8fc46c945348c88540f6217bd64/users/250596c2774b4e46a76c9dcb2b619fc6/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:26.405305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.405582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.405855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.435195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] MFA Rules not processed for user `250596c2774b4e46a76c9dcb2b619fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:26.436589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:26.519946 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:26.526091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1291/2593] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:26.539685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:26.541489 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:26.600609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1dc892-429e-4421-9805-91bdf80e6d02 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:26.604503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:26.604956 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:26.605229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:26.605497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:26.613869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:26.615655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62336801-94b3-47f1-b703-b2b23c503189 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:26.624480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1303/2594] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:27.497954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:27.499442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:27.572574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494e2c83-7592-4e7d-a363-5b7989a316b9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:27.577689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:27.578309 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:27.578602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:27.578872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] PATH_INFO: `/v3/users/7bb9375a21dc46e2b45b84214a2a269a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:27.580677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bb9375a21dc46e2b45b84214a2a269a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:27.582666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:27.633064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bb9375a21dc46e2b45b84214a2a269a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:27.640502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bb9375a21dc46e2b45b84214a2a269a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:27.653897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bb9375a21dc46e2b45b84214a2a269a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:27.654653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bb9375a21dc46e2b45b84214a2a269a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:27.655259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bb9375a21dc46e2b45b84214a2a269a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:27.669773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1292/2595] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:27 2022] DELETE /identity/v3/users/7bb9375a21dc46e2b45b84214a2a269a => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:27.674761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:27.675238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:27.675367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:27.694809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:27.696024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:27.709464 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:27.761832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1304/2596] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:27.772319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:27.773706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:27.843410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cba7f2-de86-4a64-a384-697b1287e7df admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:27.847981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:27.848577 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:27.848846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:27.849108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:27.858798 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:27.860253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:27.871115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1293/2597] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:28.038237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:28.040274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:28.120200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e2fdb2-5577-495c-ae80-09606d7eafcc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:28.124391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:28.125009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:28.125287 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:28.125549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:28.129917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:28.205928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:28.208526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:28.219440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1305/2598] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:28.226246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:28.228343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:28.262869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f774bbce-5edb-438c-bead-21179357da95 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:28.268485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:28.269402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:28.269856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:28.270138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] PATH_INFO: `/v3/projects/aebd23a226e44b09abfba4ea35808192` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:28.271595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aebd23a226e44b09abfba4ea35808192)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:28.273444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:28.291128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:28.331483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aebd23a226e44b09abfba4ea35808192', 'request_id': 'req-9eb227db-b8e3-4036-b397-2a8c2c421851'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.338739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aebd23a226e44b09abfba4ea35808192', 'request_id': 'req-9eb227db-b8e3-4036-b397-2a8c2c421851'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.347979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aebd23a226e44b09abfba4ea35808192', 'request_id': 'req-9eb227db-b8e3-4036-b397-2a8c2c421851'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.359997 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] The token cache is being invalidate because project aebd23a226e44b09abfba4ea35808192 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:28.360360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.361004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.363669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1294/2599] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:28 2022] DELETE /identity/v3/projects/aebd23a226e44b09abfba4ea35808192 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:28.368509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:28.380613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e377b7-2ba0-42b7-a629-0a5f1ce1f52d service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:28.447399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:28.448118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:28.448403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:28.448919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] PATH_INFO: `/v3/users/30024b364be54f78949a6761535b8f0f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:28.450003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30024b364be54f78949a6761535b8f0f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:28.452126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:28.472194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:28.507368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30024b364be54f78949a6761535b8f0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.514875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30024b364be54f78949a6761535b8f0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.528627 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30024b364be54f78949a6761535b8f0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.529399 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30024b364be54f78949a6761535b8f0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.538395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-53679c5d-f731-4ac8-a0c3-4ca3cbd00869 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30024b364be54f78949a6761535b8f0f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:28.544719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1306/2600] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:28 2022] DELETE /identity/v3/users/30024b364be54f78949a6761535b8f0f => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:28.839066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:28.841212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:28.856831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb227db-b8e3-4036-b397-2a8c2c421851 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:28.936714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:28.937593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:28.939072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:28.942346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] PATH_INFO: `/v3/projects/fd79204541a74073afd02f5fc7ae60d0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:28.945213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd79204541a74073afd02f5fc7ae60d0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:28.948096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.000139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd79204541a74073afd02f5fc7ae60d0', 'request_id': 'req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.006432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd79204541a74073afd02f5fc7ae60d0', 'request_id': 'req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.014434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd79204541a74073afd02f5fc7ae60d0', 'request_id': 'req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.025212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] The token cache is being invalidate because project fd79204541a74073afd02f5fc7ae60d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:29.025490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.026108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad16dc66-b2ec-4708-82d8-beb6ff5914e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.027615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1295/2601] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:28 2022] DELETE /identity/v3/projects/fd79204541a74073afd02f5fc7ae60d0 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:29.032510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.033051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.033685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.055487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:29.056674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.124089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1307/2602] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:29.132436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.132715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.132982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.155432 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:29.157078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.165846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1296/2603] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:29.170241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.180190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365ed0f3-503a-4c8f-ad54-8c7d1fbb5949 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.184699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.185296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.185570 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.185839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.186486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:29.186812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.188407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.200608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1308/2604] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:29.205977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.215211 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2dd5e3-325b-4abe-a71b-e777e80ee75e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.219990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.220767 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.221568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.221929 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.222740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.224990 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.255834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1297/2605] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:29.261812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.272794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df08f1a-d8f5-4b24-8866-509b50ae8af2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.274426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.276123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.282402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:29.340177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.340989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.341265 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.341547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.342186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.344134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.347072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc21dea4-8cf5-4ce5-864c-a3c6febcf438 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.351704 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.352333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.352612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.352883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] PATH_INFO: `/v3/users/250596c2774b4e46a76c9dcb2b619fc6` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.354213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] RBAC: Authorizing `identity:delete_user(user_id=250596c2774b4e46a76c9dcb2b619fc6)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.356199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.367200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1309/2606] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/users => generated 289 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:29.371456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.382507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594ff3f4-aefa-4a73-a299-9888539fcf5b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.388054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.388600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.389019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.389324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.390130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.392296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.399395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:29.406603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1310/2607] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:29.409745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '250596c2774b4e46a76c9dcb2b619fc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.410092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.416928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '250596c2774b4e46a76c9dcb2b619fc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.420904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157fee29-8994-4d2a-bfda-859be6a985f1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.426006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.426622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.426905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.427171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.427804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.428888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '250596c2774b4e46a76c9dcb2b619fc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.429385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.429582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '250596c2774b4e46a76c9dcb2b619fc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.430191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4616e436-d1b8-45ad-88ea-2282545e7330 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '250596c2774b4e46a76c9dcb2b619fc6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:29.441723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1311/2608] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:29.445772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.446034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1298/2609] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:29 2022] DELETE /identity/v3/users/250596c2774b4e46a76c9dcb2b619fc6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:29.447688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.450348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.450681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.451099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.462245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db34d095-4c65-4297-b26e-e1674f87f046 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.471167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:29.472381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.521996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.522697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.522979 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.523255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] PATH_INFO: `/v3/projects/a6b3f80220af4a36b785a8cd7408c923/users/b156fba8672846ae870a08ab9779611b/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.540186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6b3f80220af4a36b785a8cd7408c923, user_id=b156fba8672846ae870a08ab9779611b, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.541516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1299/2610] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:29.545629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02221ff-a112-4635-a2f7-9bada54268ba admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.552107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.553728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.559830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1312/2611] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:29 2022] PUT /identity/v3/projects/a6b3f80220af4a36b785a8cd7408c923/users/b156fba8672846ae870a08ab9779611b/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:29.571691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.572223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.572223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.585830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:29.607345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] MFA Rules not processed for user `b156fba8672846ae870a08ab9779611b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:29.608766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:29.619517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28f35c-ec8a-438d-bd3e-94248b8f0646 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.674122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.675112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.675438 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.675708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.679147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1313/2612] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:29.683716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.760474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.761921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.767578 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231e6748-5990-46e4-92c4-294ac81dfa46 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.772760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1300/2613] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:54:29.773776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.773776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.774145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.774952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.775105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.777028 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.783114 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:29.813077 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1314/2614] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/projects => generated 377 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:29.819039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.838776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0644a75-72b4-42f6-ad6c-12f0fab07510 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.909119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.909981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.910364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.911336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.912032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.914430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.940010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1301/2615] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:29 2022] POST /identity/v3/users => generated 295 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:29.945448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:29.957226 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58273012-2305-4f6d-9c53-62e043882f5b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:29.962771 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:29.963666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:29.963960 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:29.964273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:29.965225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:29.967351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:29.996780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1315/2616] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:29.997523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.009347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f3828-5c46-48be-a74a-fe20f3743054 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.016047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.016760 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.017034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.017460 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] PATH_INFO: `/v3/projects/fa933e312edf4d2f9b42e949c980da54/users/76f8f25a0af04661b7517afb00996cec/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.037103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa933e312edf4d2f9b42e949c980da54, user_id=76f8f25a0af04661b7517afb00996cec, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.039838 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.044016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:30.059680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1302/2617] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:29 2022] PUT /identity/v3/projects/fa933e312edf4d2f9b42e949c980da54/users/76f8f25a0af04661b7517afb00996cec/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:30.063669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.076385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd207291-e1e6-4d12-8093-b6129eb270dd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.183215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.185191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.186092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.186793 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.187118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.187406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.188142 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.190689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.202537 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae4c607-f71a-4303-8455-607f354ffcf1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.206613 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1316/2618] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:30 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:30.210248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.226723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.230984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.230984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.230984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] PATH_INFO: `/v3/projects/bdf5f8fc46c945348c88540f6217bd64` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.230984 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf5f8fc46c945348c88540f6217bd64)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.231343 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27462c-b538-4067-a9bf-a36337477fcc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.232561 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.242316 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.243273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.243935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.244300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] PATH_INFO: `/v3/projects/fa933e312edf4d2f9b42e949c980da54/users/76f8f25a0af04661b7517afb00996cec/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.247867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa933e312edf4d2f9b42e949c980da54, user_id=76f8f25a0af04661b7517afb00996cec, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.254871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a56bf2-6e49-40e6-97e6-b654b52bd035 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.277360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1317/2619] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:30 2022] PUT /identity/v3/projects/fa933e312edf4d2f9b42e949c980da54/users/76f8f25a0af04661b7517afb00996cec/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:30.290708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.291199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.291642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.301470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf5f8fc46c945348c88540f6217bd64', 'request_id': 'req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.308446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf5f8fc46c945348c88540f6217bd64', 'request_id': 'req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.320036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf5f8fc46c945348c88540f6217bd64', 'request_id': 'req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.336743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] The token cache is being invalidate because project bdf5f8fc46c945348c88540f6217bd64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:30.337175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.338168 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.339653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] MFA Rules not processed for user `76f8f25a0af04661b7517afb00996cec`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:30.339795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1303/2620] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:30 2022] DELETE /identity/v3/projects/bdf5f8fc46c945348c88540f6217bd64 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:30.342097 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.437840 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1318/2621] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:30 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:30.451927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.453972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.549393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b5bfb4-157a-4d80-bf03-3cb33cf98a3e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.554810 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.555475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.555763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.556043 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.557958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.567404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.569398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.628375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:30.633601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.635315 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6533dd56-2166-4082-9691-027fc87ee5b5 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.645277 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1304/2622] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:30.651061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44953d13-d214-4170-9d5f-0911d75b8f3f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.656216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.656912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.657351 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.657630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] PATH_INFO: `/v3/users/5b7c7805d26443f8ae443555d5e1dd2f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.660602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b7c7805d26443f8ae443555d5e1dd2f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.662637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.721386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b7c7805d26443f8ae443555d5e1dd2f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.728565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b7c7805d26443f8ae443555d5e1dd2f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.742977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b7c7805d26443f8ae443555d5e1dd2f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.744002 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b7c7805d26443f8ae443555d5e1dd2f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.751890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b7c7805d26443f8ae443555d5e1dd2f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:30.758923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1319/2623] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:30 2022] DELETE /identity/v3/users/5b7c7805d26443f8ae443555d5e1dd2f => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:30.763555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.763836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.764130 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.784628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:30.785833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.815562 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:30.851619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1305/2624] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:30.861944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:30.863631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:30.945940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e09273-88aa-4945-85a5-bb8f6a03fd13 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:30.950217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:30.951055 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:30.951485 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:30.951813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:30.966601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:30.966601 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:30.977905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1320/2625] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:31.271976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:31.273555 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:31.293739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b2d6de-e3cd-46f0-aed5-6869ead1e3dc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:31.306848 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:31.307529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:31.307802 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:31.308069 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] PATH_INFO: `/v3/projects/ceeb8683cf0445dca652d70cd9782007` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:31.309195 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ceeb8683cf0445dca652d70cd9782007)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:31.311142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:31.361507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ceeb8683cf0445dca652d70cd9782007', 'request_id': 'req-d563c888-ec0e-4717-ae66-ed97bb791215'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.369392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ceeb8683cf0445dca652d70cd9782007', 'request_id': 'req-d563c888-ec0e-4717-ae66-ed97bb791215'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.377859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ceeb8683cf0445dca652d70cd9782007', 'request_id': 'req-d563c888-ec0e-4717-ae66-ed97bb791215'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.388606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] The token cache is being invalidate because project ceeb8683cf0445dca652d70cd9782007 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:31.388893 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.389523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.391059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1306/2626] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:31 2022] DELETE /identity/v3/projects/ceeb8683cf0445dca652d70cd9782007 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:31.395179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:31.404839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b1526-be97-4a5b-946c-868432233193 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:31.462648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:31.463273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:31.463544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:31.463817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] PATH_INFO: `/v3/users/773543a5fe054e7a83af91582055ff0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:31.464913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=773543a5fe054e7a83af91582055ff0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:31.466858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:31.472536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:31.518749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '773543a5fe054e7a83af91582055ff0e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.525511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '773543a5fe054e7a83af91582055ff0e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.537141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '773543a5fe054e7a83af91582055ff0e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.537834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '773543a5fe054e7a83af91582055ff0e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.545334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '773543a5fe054e7a83af91582055ff0e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.552674 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1321/2627] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:31 2022] DELETE /identity/v3/users/773543a5fe054e7a83af91582055ff0e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:31.842741 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:31.844638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:31.862269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563c888-ec0e-4717-ae66-ed97bb791215 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:31.927449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:31.928090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:31.928364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:31.928635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] PATH_INFO: `/v3/projects/63ea233e626042c7a2f01ff2f9777161` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:31.929759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63ea233e626042c7a2f01ff2f9777161)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:31.931699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:31.979816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63ea233e626042c7a2f01ff2f9777161', 'request_id': 'req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.986763 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63ea233e626042c7a2f01ff2f9777161', 'request_id': 'req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:31.991247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:31.996608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63ea233e626042c7a2f01ff2f9777161', 'request_id': 'req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.008718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] The token cache is being invalidate because project 63ea233e626042c7a2f01ff2f9777161 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:32.009006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.009721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.011672 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1307/2628] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:31 2022] DELETE /identity/v3/projects/63ea233e626042c7a2f01ff2f9777161 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:32.015685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.017135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:32.031069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f587fc5-106b-496d-94eb-f9304b5e31c9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.090625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.091288 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.091565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.091835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] PATH_INFO: `/v3/users/13869e3d6e374fd8812b5e44bc0ac3f2` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.092987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13869e3d6e374fd8812b5e44bc0ac3f2)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.094945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.164449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13869e3d6e374fd8812b5e44bc0ac3f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.171614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13869e3d6e374fd8812b5e44bc0ac3f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.186063 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13869e3d6e374fd8812b5e44bc0ac3f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.186847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13869e3d6e374fd8812b5e44bc0ac3f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.195998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0bc03786-1b2e-4f05-806d-e3097be97d70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13869e3d6e374fd8812b5e44bc0ac3f2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.206871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1322/2629] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:32 2022] DELETE /identity/v3/users/13869e3d6e374fd8812b5e44bc0ac3f2 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:32.492531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.494243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:32.508602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b1b5a-662a-49e0-9b6d-5693aab2ffa7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.572222 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.572764 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.573037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.573303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] PATH_INFO: `/v3/projects/fd6650884b42467795f08b4f3f849457` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.574427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd6650884b42467795f08b4f3f849457)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.576331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.621856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd6650884b42467795f08b4f3f849457', 'request_id': 'req-a1a67923-20d1-4b87-8d8f-9a686c9c370e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.628677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd6650884b42467795f08b4f3f849457', 'request_id': 'req-a1a67923-20d1-4b87-8d8f-9a686c9c370e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.636710 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd6650884b42467795f08b4f3f849457', 'request_id': 'req-a1a67923-20d1-4b87-8d8f-9a686c9c370e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.647547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] The token cache is being invalidate because project fd6650884b42467795f08b4f3f849457 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:32.647832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.648487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a1a67923-20d1-4b87-8d8f-9a686c9c370e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:32.649959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1308/2630] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:32 2022] DELETE /identity/v3/projects/fd6650884b42467795f08b4f3f849457 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:32.656144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.656648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.656744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.665841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:32.677504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:32.678757 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:32.738373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1323/2631] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:32.747050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.747328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.747620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.768033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:32.769232 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:32.777067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1309/2632] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:32.781647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.792448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1993c5-abca-4f15-ae61-4b1f867e020e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.797244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.797854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.798127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.798981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.799798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:32.800099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.801770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.815006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1324/2633] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:54:32.819600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.828753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd0c990-c97c-48e2-b597-22516e70e88c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.833878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.834473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.834823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.835090 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.835757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.837625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.868082 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1310/2634] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:32 2022] POST /identity/v3/projects => generated 389 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:32.872465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.883176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfaa7f3-d8e7-491d-b2b2-580a35586611 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.938939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.940182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.940439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.940703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.941328 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.943276 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.965136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1325/2635] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:32 2022] POST /identity/v3/users => generated 295 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:32.969317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:32.980386 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f062cb4-7574-4bc4-a50d-6ed36b8e64d2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:32.985004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:32.985595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:32.986632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:32.986995 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:32.987642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:32.989463 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:32.999283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1311/2636] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:33.004342 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.016240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b714e4-98bc-4928-b5c9-a977cbd41baa admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.025495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.026261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.026579 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.026863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.027529 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.029169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad492a1-cbbc-4622-bae6-fdfb77e573c2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.042236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1326/2637] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:33.046709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.057486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c544d2-cde8-4b41-93d5-5a8c8eb9f5f6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.062050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.062715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.062988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.063249 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] PATH_INFO: `/v3/projects/40fbafa808dd4532bbe2db447b67b588/users/3fe302a523ea47b5a1fa214bd91f6a9f/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.078296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40fbafa808dd4532bbe2db447b67b588, user_id=3fe302a523ea47b5a1fa214bd91f6a9f, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.081066 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.095411 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1312/2638] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:33 2022] PUT /identity/v3/projects/40fbafa808dd4532bbe2db447b67b588/users/3fe302a523ea47b5a1fa214bd91f6a9f/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:33.107683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.107971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.108250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.138730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] MFA Rules not processed for user `3fe302a523ea47b5a1fa214bd91f6a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:33.139978 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:33.202701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:33.207595 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1327/2639] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:33 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:33.212184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.229120 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:33.288727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb57fc3-df8d-4f48-9603-94abbb199d45 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.294372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.295013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.295317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.295585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.296218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.298111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.327859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1313/2640] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:33 2022] POST /identity/v3/projects => generated 389 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:33.332309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.342865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5bfe0e-2b8e-4922-9c77-446a7476493a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.407228 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.408486 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.408905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.409408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.410105 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.412062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.440623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1328/2641] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:33 2022] POST /identity/v3/users => generated 301 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:33.444894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.455694 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd0531-65fa-4207-b68d-bd5a20c6eacc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.460628 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.461139 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.461410 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.461702 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.462374 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.463970 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.475470 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1314/2642] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:33.480210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.491640 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d49571-c5f0-49a0-aaac-5fe6593aee25 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.496419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.497106 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.497378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.497648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] PATH_INFO: `/v3/projects/2084ab825e8b4cbf986c5717b1d57f94/users/7e53b205eb4c4c8b88a43a78d34c78fd/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.514902 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2084ab825e8b4cbf986c5717b1d57f94, user_id=7e53b205eb4c4c8b88a43a78d34c78fd, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.518217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.532227 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1329/2643] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:33 2022] PUT /identity/v3/projects/2084ab825e8b4cbf986c5717b1d57f94/users/7e53b205eb4c4c8b88a43a78d34c78fd/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:33.537017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.547193 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4a831-106f-46fb-98c9-506b953759af admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.604121 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.604656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.605030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.605296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.605940 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.607591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80de8c24-08db-4c77-a016-ce7f2d26f6e5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.617755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1315/2644] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:33 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:33.622044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.632371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fc53d-f3a3-41b6-ab85-49b5abe8f11a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.636912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.637557 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.637831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.638100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] PATH_INFO: `/v3/projects/2084ab825e8b4cbf986c5717b1d57f94/users/7e53b205eb4c4c8b88a43a78d34c78fd/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.640001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2084ab825e8b4cbf986c5717b1d57f94, user_id=7e53b205eb4c4c8b88a43a78d34c78fd, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.643582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.654245 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1330/2645] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:33 2022] PUT /identity/v3/projects/2084ab825e8b4cbf986c5717b1d57f94/users/7e53b205eb4c4c8b88a43a78d34c78fd/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:33.666094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.666368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.666676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.696123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] MFA Rules not processed for user `7e53b205eb4c4c8b88a43a78d34c78fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:33.697350 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:33.767048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1316/2646] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:33 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:33.778744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:33.780465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:33.837883 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d33292-fc19-4e0d-a2ed-2c06488f45e2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:33.841562 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:33.842165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:33.842445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:33.842744 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:33.852406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:33.853864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850f1af8-f77e-4774-b816-f9b90da52974 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:33.863510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1331/2647] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:33 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:34.004976 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:34.015586 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:34.081296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bbfe95-9276-4b44-aa7e-e56f25206dd4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:34.087471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:34.088095 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.088365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.088636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] PATH_INFO: `/v3/users/3fe302a523ea47b5a1fa214bd91f6a9f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.089742 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fe302a523ea47b5a1fa214bd91f6a9f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:34.091669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:34.144787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fe302a523ea47b5a1fa214bd91f6a9f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.153866 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fe302a523ea47b5a1fa214bd91f6a9f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.168302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fe302a523ea47b5a1fa214bd91f6a9f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.169050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fe302a523ea47b5a1fa214bd91f6a9f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.169631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fe302a523ea47b5a1fa214bd91f6a9f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.182720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1317/2648] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:34 2022] DELETE /identity/v3/users/3fe302a523ea47b5a1fa214bd91f6a9f => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:34.187714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.187999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.188273 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.208300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:34.209504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:34.258049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:34.270498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1332/2649] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:34.280761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:34.282078 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:34.342610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9b295d-9599-419c-a28b-c76072947d42 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:34.346370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:34.347029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.347305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.347591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.357697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:34.359263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:34.371227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1318/2650] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:34.676279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:34.677630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:34.696496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de07137-72eb-4628-9ec9-dcb9eb2421b8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:34.703669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:34.704649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.706449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.706449 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] PATH_INFO: `/v3/projects/40fbafa808dd4532bbe2db447b67b588` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.707547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40fbafa808dd4532bbe2db447b67b588)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:34.710261 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:34.765950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40fbafa808dd4532bbe2db447b67b588', 'request_id': 'req-df84a222-7cf1-42a7-b2d7-974e0492f186'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.773479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40fbafa808dd4532bbe2db447b67b588', 'request_id': 'req-df84a222-7cf1-42a7-b2d7-974e0492f186'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.783478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40fbafa808dd4532bbe2db447b67b588', 'request_id': 'req-df84a222-7cf1-42a7-b2d7-974e0492f186'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.800207 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] The token cache is being invalidate because project 40fbafa808dd4532bbe2db447b67b588 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:34.800532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.801194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.802733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1333/2651] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:34 2022] DELETE /identity/v3/projects/40fbafa808dd4532bbe2db447b67b588 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:34.807098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:34.817270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf736fd-363a-422d-ba8e-6dd7f9a77dc6 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:34.847048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:34.877827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:34.878444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:34.878743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:34.879007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] PATH_INFO: `/v3/users/7e53b205eb4c4c8b88a43a78d34c78fd` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:34.880115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e53b205eb4c4c8b88a43a78d34c78fd)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:34.881998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:34.926092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e53b205eb4c4c8b88a43a78d34c78fd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.932830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e53b205eb4c4c8b88a43a78d34c78fd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.945491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e53b205eb4c4c8b88a43a78d34c78fd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.946169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e53b205eb4c4c8b88a43a78d34c78fd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.946801 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e53b205eb4c4c8b88a43a78d34c78fd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:34.960725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1319/2652] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:34 2022] DELETE /identity/v3/users/7e53b205eb4c4c8b88a43a78d34c78fd => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:35.254224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.255684 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:35.270869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df84a222-7cf1-42a7-b2d7-974e0492f186 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.346376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.348174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:35.349237 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.350391 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.350527 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.350813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] PATH_INFO: `/v3/projects/2084ab825e8b4cbf986c5717b1d57f94` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.352079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2084ab825e8b4cbf986c5717b1d57f94)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.359036 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.416280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2084ab825e8b4cbf986c5717b1d57f94', 'request_id': 'req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:35.422241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799e37e-e28b-4cfa-a254-df17715b2d0e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.423205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2084ab825e8b4cbf986c5717b1d57f94', 'request_id': 'req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:35.433078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2084ab825e8b4cbf986c5717b1d57f94', 'request_id': 'req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:35.445007 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] The token cache is being invalidate because project 2084ab825e8b4cbf986c5717b1d57f94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:35.445647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:35.446480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b47ff90b-0ee6-44c6-acdd-e9a960b6ede0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:35.448166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1334/2653] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:35 2022] DELETE /identity/v3/projects/2084ab825e8b4cbf986c5717b1d57f94 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:35.454121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed4a95-5caa-4c50-aa4d-512eedf4d6c8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.454262 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed4a95-5caa-4c50-aa4d-512eedf4d6c8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.455335 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed4a95-5caa-4c50-aa4d-512eedf4d6c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.477958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-79ed4a95-5caa-4c50-aa4d-512eedf4d6c8 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:35.479620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-79ed4a95-5caa-4c50-aa4d-512eedf4d6c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:35.484037 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:35.490898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.491524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.491859 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.492064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.494102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:35.521969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:35.549532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1335/2654] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:35.557680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.557964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.558233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.575758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.577147 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.577829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:35.579044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:35.587387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1320/2655] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:35 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:35.589855 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1336/2656] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:54:35.594219 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.604952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ad8a4f-7ad2-41f1-b527-9ebbcdc40510 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.609393 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.610040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.610346 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.610634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.611578 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:35.611876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.613509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.625338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1321/2657] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:54:35.630519 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.640218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb5f832-a348-4f3a-ac86-2462a8383423 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.645543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.646150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.646430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.646723 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.647349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.649268 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.677336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1337/2658] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:35 2022] POST /identity/v3/projects => generated 379 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:35.682673 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.693513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f721820e-fdd9-4f74-b7f4-4c355624193e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.770107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.770975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.771253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.771528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.772159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.774126 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.806785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1322/2659] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:35 2022] POST /identity/v3/users => generated 290 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:35.811259 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.822496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a0080-8df0-46e9-8565-f2aebbdd45cb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.827734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.828250 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.828526 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.828796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.829467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.831482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.842906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1338/2660] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:35.847186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.858498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2921b7b-ca0a-4eed-b163-c62ca16a70f6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.865143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.865800 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.866079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.866485 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.867047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.868661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8537eb4e-110a-4fee-b778-ef9f172fce47 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.880784 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1323/2661] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:35.885799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:35.897530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71169ad-2adf-4675-8b6a-2b0a2261efc2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:35.902862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:35.903538 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:35.903812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:35.904079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] PATH_INFO: `/v3/projects/f70df2303afd45009aeaa0c205cbbf13/users/b761af7e623c4218b427a53eea07eaad/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:35.920830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f70df2303afd45009aeaa0c205cbbf13, user_id=b761af7e623c4218b427a53eea07eaad, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:35.924765 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:35.951872 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1339/2662] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:35 2022] PUT /identity/v3/projects/f70df2303afd45009aeaa0c205cbbf13/users/b761af7e623c4218b427a53eea07eaad/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:36.037974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.038273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.038502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.070785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] MFA Rules not processed for user `b761af7e623c4218b427a53eea07eaad`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:36.072027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:36.149488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1324/2663] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:36.154217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.218430 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eba906-8249-4d92-afe8-cef5177a2b09 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.224153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.224787 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.225107 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.225353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.226000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.227989 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.260085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1340/2664] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/projects => generated 377 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:36.264132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.275370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c65847b-4240-4de6-884e-d7d6d98ec9d5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.283509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:36.336054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.336661 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.336932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.337197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.337819 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.339739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.362975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1325/2665] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:36.367543 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.378308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b3dc3-b12b-4e84-a25a-bbdc00210308 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.383153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.383750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.384025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.384290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.385014 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.386725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.397946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1341/2666] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:36.402405 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.413113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1d9e1-7969-4474-a641-d46e8122f43e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.417754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.418394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.418711 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.419004 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.419608 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.421194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa4f38-4bce-4b84-82b2-8599490fe0ce admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.432347 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1326/2667] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:36.437275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.449090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047e531f-20da-4ad3-9368-fd4daf86055d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.453761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.454405 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.454880 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.455203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] PATH_INFO: `/v3/projects/a03ed6cfa380481f84c58391c6f5e8bd/users/ef36c6cace0f44ef9f936f78049dd672/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.469676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a03ed6cfa380481f84c58391c6f5e8bd, user_id=ef36c6cace0f44ef9f936f78049dd672, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.472877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.476296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:36.489794 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1342/2668] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:36 2022] PUT /identity/v3/projects/a03ed6cfa380481f84c58391c6f5e8bd/users/ef36c6cace0f44ef9f936f78049dd672/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:36.500241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.500512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.500860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.531897 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] MFA Rules not processed for user `ef36c6cace0f44ef9f936f78049dd672`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:36.533889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:36.603919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1327/2669] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:36.608566 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.677969 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e27ebdc-72c7-41d5-bb80-2b6e7a434d3f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.683083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.683709 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.683984 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.684260 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.684926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.686881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.717470 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1343/2670] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/projects => generated 377 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:36.722634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.734239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dd3b76-9f65-48db-a0eb-834a76ccdcbe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.802142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.802790 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.803103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.803376 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.804061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.805986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.828103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1328/2671] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:36 2022] POST /identity/v3/users => generated 295 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:54:36.833385 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.844284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0175dc-49cc-491c-b1f2-897174526811 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.850026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.850667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.850957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.851233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.851915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.853531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.865753 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1344/2672] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:36.870909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.884861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a79ba-2607-4d84-9215-6dfab8099d7e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:36.893071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:36.893737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:36.894056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:36.894330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] PATH_INFO: `/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921/users/3e18d6675c0640a986ede0849808c476/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:36.897856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:36.912272 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcadfbc8ef71423bbd86b58c7a3d4921, user_id=3e18d6675c0640a986ede0849808c476, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:36.915041 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:36.930827 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1329/2673] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:36 2022] PUT /identity/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921/users/3e18d6675c0640a986ede0849808c476/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:54:36.935070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:36.946749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927810c-33f1-48c1-b3a4-0213ad92c988 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:37.011256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:37.011874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:37.012158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:37.012425 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:37.013100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:37.014748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cba157-6a79-4c09-9f9d-7ca6c91bd4be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:37.032375 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1345/2674] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:36 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:37.035484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:37.046691 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d26c88-9f14-4566-9ffe-46a21b52d5b2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:37.051926 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:37.052882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:37.053182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:37.053457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] PATH_INFO: `/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921/users/3e18d6675c0640a986ede0849808c476/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:37.055476 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcadfbc8ef71423bbd86b58c7a3d4921, user_id=3e18d6675c0640a986ede0849808c476, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:37.058115 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:37.072436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1330/2675] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:37 2022] PUT /identity/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921/users/3e18d6675c0640a986ede0849808c476/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:37.087909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:37.088165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:37.088460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:37.118467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] MFA Rules not processed for user `3e18d6675c0640a986ede0849808c476`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:37.119942 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:37.186154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:37.209217 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1346/2676] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:37 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:37.221163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:37.223030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:37.285449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c9ecd1-5963-40fa-af0e-635efab6ea8f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:37.289205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:37.289833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:37.290116 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:37.290383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:37.300857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:37.302314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:37.314772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1331/2677] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:37 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:37.362143 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:37.364699 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:37.378078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b7e68b-ffe2-48fe-9c96-b2f6ef597c56 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:37.383165 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:37.384213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:37.384691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:37.385163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:37.605796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:37.607547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde0e03a-d5be-4b29-9676-b5b916f4d7bc service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:37.620068 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1347/2678] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:41.853624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:41.918636 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e91cf65-dbdb-4df9-8439-d6a0c2997445 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:41.923305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:41.924031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:41.924313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:41.924583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] PATH_INFO: `/v3/users/b761af7e623c4218b427a53eea07eaad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:41.925743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b761af7e623c4218b427a53eea07eaad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:41.927700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:41.991191 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b761af7e623c4218b427a53eea07eaad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:41.998314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b761af7e623c4218b427a53eea07eaad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.010729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b761af7e623c4218b427a53eea07eaad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.011450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b761af7e623c4218b427a53eea07eaad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.012072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b761af7e623c4218b427a53eea07eaad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.014856 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:42.029390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1332/2679] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:41 2022] DELETE /identity/v3/users/b761af7e623c4218b427a53eea07eaad => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:42.034735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:42.035387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:42.035387 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:42.055384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:42.056770 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:42.119644 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1348/2680] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:42.129975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:42.131439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:42.189814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719ed6b-217d-491c-92b3-b6425f89f980 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:42.193459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:42.194061 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:42.194334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:42.194633 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:42.203509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:42.205517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:42.214842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1333/2681] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:42.485027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:42.487406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:42.500067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514de0c6-f456-49cf-ab87-6de5c0ec8cc0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:42.508846 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:42.509531 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:42.510156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:42.510156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] PATH_INFO: `/v3/projects/f70df2303afd45009aeaa0c205cbbf13` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:42.511277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=f70df2303afd45009aeaa0c205cbbf13)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:42.513433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:42.576219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f70df2303afd45009aeaa0c205cbbf13', 'request_id': 'req-324c55e7-4d86-43c7-b2fc-17e965f408ad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.585916 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f70df2303afd45009aeaa0c205cbbf13', 'request_id': 'req-324c55e7-4d86-43c7-b2fc-17e965f408ad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.595987 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f70df2303afd45009aeaa0c205cbbf13', 'request_id': 'req-324c55e7-4d86-43c7-b2fc-17e965f408ad'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.606775 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] The token cache is being invalidate because project f70df2303afd45009aeaa0c205cbbf13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:42.607066 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.607722 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.609210 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1349/2682] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:42 2022] DELETE /identity/v3/projects/f70df2303afd45009aeaa0c205cbbf13 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:42.613816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:42.623849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6990b81b-3e20-48ad-bb44-7b69a50a0ca8 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:42.684099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:42.684731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:42.685034 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:42.685305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] PATH_INFO: `/v3/users/ef36c6cace0f44ef9f936f78049dd672` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:42.686431 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef36c6cace0f44ef9f936f78049dd672)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:42.688738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:42.734714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef36c6cace0f44ef9f936f78049dd672'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.742950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef36c6cace0f44ef9f936f78049dd672'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.757079 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef36c6cace0f44ef9f936f78049dd672'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.757849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef36c6cace0f44ef9f936f78049dd672'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.758520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef36c6cace0f44ef9f936f78049dd672'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:42.784390 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1334/2683] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:42 2022] DELETE /identity/v3/users/ef36c6cace0f44ef9f936f78049dd672 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:43.088264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:43.089839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:43.111865 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324c55e7-4d86-43c7-b2fc-17e965f408ad admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:43.123957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:43.185740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:43.186503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:43.186898 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:43.187275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] PATH_INFO: `/v3/projects/a03ed6cfa380481f84c58391c6f5e8bd` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:43.188540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a03ed6cfa380481f84c58391c6f5e8bd)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:43.190923 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:43.242478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a03ed6cfa380481f84c58391c6f5e8bd', 'request_id': 'req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.249021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a03ed6cfa380481f84c58391c6f5e8bd', 'request_id': 'req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.258617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a03ed6cfa380481f84c58391c6f5e8bd', 'request_id': 'req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.271434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] The token cache is being invalidate because project a03ed6cfa380481f84c58391c6f5e8bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:43.271796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.272599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.274346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1350/2684] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:43 2022] DELETE /identity/v3/projects/a03ed6cfa380481f84c58391c6f5e8bd => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:43.278436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:43.279989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:43.293443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8738777-97aa-4025-ba97-0b0c14dea698 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:43.310682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:43.362836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:43.363481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:43.363752 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:43.364287 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] PATH_INFO: `/v3/users/3e18d6675c0640a986ede0849808c476` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:43.365465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e18d6675c0640a986ede0849808c476)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:43.367465 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:43.428296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e18d6675c0640a986ede0849808c476'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.436333 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e18d6675c0640a986ede0849808c476'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.453057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e18d6675c0640a986ede0849808c476'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.453799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e18d6675c0640a986ede0849808c476'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.455104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9fce7c1-82bc-499f-adb3-822651875f30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e18d6675c0640a986ede0849808c476'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.471123 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1335/2685] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:54:43 2022] DELETE /identity/v3/users/3e18d6675c0640a986ede0849808c476 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:43.779558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:43.781000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:43.794224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dbf429-7399-4dbc-8b98-ac330d8b63e1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:43.857166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:43.857816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:43.858095 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:43.858367 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] PATH_INFO: `/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:43.859535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcadfbc8ef71423bbd86b58c7a3d4921)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:43.861747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:43.914525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:43.919086 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcadfbc8ef71423bbd86b58c7a3d4921', 'request_id': 'req-56d3aec6-cd63-4b42-910d-20a9e740d776'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.927834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcadfbc8ef71423bbd86b58c7a3d4921', 'request_id': 'req-56d3aec6-cd63-4b42-910d-20a9e740d776'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.937533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcadfbc8ef71423bbd86b58c7a3d4921', 'request_id': 'req-56d3aec6-cd63-4b42-910d-20a9e740d776'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.953886 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] The token cache is being invalidate because project dcadfbc8ef71423bbd86b58c7a3d4921 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:54:43.954334 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.956109 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-56d3aec6-cd63-4b42-910d-20a9e740d776 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:54:43.958370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1351/2686] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:54:43 2022] DELETE /identity/v3/projects/dcadfbc8ef71423bbd86b58c7a3d4921 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:43.966870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:43.967216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:43.967588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:43.991046 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:43.992285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:44.062949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1336/2687] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:44.073534 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.074013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.074495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.107546 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:44.109582 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:44.123477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1352/2688] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:44.128922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.139391 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7ed812-7bc1-41c4-9753-e95ece6ef84b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.144732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.145357 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.145629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.145900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.146574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:54:44.146872 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.148472 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.161988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1337/2689] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:54:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:54:44.166251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.176616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dfb0a9-37bb-4597-a5fd-ec0188de5f3a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.181630 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.182266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.182540 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.182834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.183468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.185408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.214255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1353/2690] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:54:44 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:44.218509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.228918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2984057f-ad41-4739-848d-79f316f58c5d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.249022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:44.290308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.291102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.291206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.291446 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.292101 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.294027 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.318574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1338/2691] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:54:44 2022] POST /identity/v3/users => generated 285 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:54:44.322497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.333278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e22ac8-761c-4ea8-9e49-a6fabfa18a6c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.338025 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.338729 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.338999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.339219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.339893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.341504 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.353119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1354/2692] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:54:44.362518 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.373039 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1183138-5f2a-4594-9637-d7f9ad3dbfe8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.379385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.380099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.380414 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.380722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.381422 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.383157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c79915a-9c31-4f3d-a78e-d207d87d2dfb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.395830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1339/2693] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:54:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:54:44.400057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.411166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cdf8c-37f2-4b53-afcf-4c4556d26e68 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.416231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.416893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.417170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.417465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] PATH_INFO: `/v3/projects/afdcf85bf60b4105a07b7e04123f94e8/users/683fa5fa5ed14c4f9244817664e3c8be/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.434257 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afdcf85bf60b4105a07b7e04123f94e8, user_id=683fa5fa5ed14c4f9244817664e3c8be, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.437020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.458029 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1355/2694] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:54:44 2022] PUT /identity/v3/projects/afdcf85bf60b4105a07b7e04123f94e8/users/683fa5fa5ed14c4f9244817664e3c8be/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:54:44.472657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.472922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.473106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.505601 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] MFA Rules not processed for user `683fa5fa5ed14c4f9244817664e3c8be`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:54:44.506916 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:44.576889 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1340/2695] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:54:44 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:54:44.590224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:54:44.591957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:54:44.679078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce188ab3-8b47-4694-a669-ac448fa214d4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:54:44.683044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:54:44.683772 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:54:44.683940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:54:44.684209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:54:44.696745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:54:44.698322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ba4532-762c-436e-9493-07f22d95a8b9 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:54:44.709635 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1356/2696] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:54:44 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:54:54.573883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=56,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:54:54.704161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=4,keystone:SELECT=69,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:00.117067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:00.118934 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:00.212471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de39b169-097e-4c47-8ad2-a6d6a6bf2f34 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:00.222924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:00.222924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:00.222924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:00.222924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] PATH_INFO: `/v3/users/b156fba8672846ae870a08ab9779611b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:00.223851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b156fba8672846ae870a08ab9779611b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:00.225771 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:00.302097 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b156fba8672846ae870a08ab9779611b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.310458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b156fba8672846ae870a08ab9779611b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.326861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b156fba8672846ae870a08ab9779611b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.327641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b156fba8672846ae870a08ab9779611b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.328240 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b156fba8672846ae870a08ab9779611b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.343261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1341/2697] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:00 2022] DELETE /identity/v3/users/b156fba8672846ae870a08ab9779611b => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:00.348076 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:00.348605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:00.348605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:00.371828 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:00.373067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:00.450403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1357/2698] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:00.461388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:00.463150 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:00.524228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3736ca49-2d6e-4066-b14d-7fb1f386c5e7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:00.532224 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:00.532963 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:00.533235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:00.533498 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:00.546406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:00.547876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:00.558612 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1342/2699] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:00.844972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:00.846813 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:00.871186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e80fbf-c249-42df-b960-0026702c0618 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:00.875991 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:00.876631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:00.876971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:00.877247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] PATH_INFO: `/v3/projects/a6b3f80220af4a36b785a8cd7408c923` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:00.878918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6b3f80220af4a36b785a8cd7408c923)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:00.880884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:00.934274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6b3f80220af4a36b785a8cd7408c923', 'request_id': 'req-fce6a1b2-e475-4afa-9f6a-d5106934797f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.941545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6b3f80220af4a36b785a8cd7408c923', 'request_id': 'req-fce6a1b2-e475-4afa-9f6a-d5106934797f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.950061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6b3f80220af4a36b785a8cd7408c923', 'request_id': 'req-fce6a1b2-e475-4afa-9f6a-d5106934797f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.961895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] The token cache is being invalidate because project a6b3f80220af4a36b785a8cd7408c923 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:00.962191 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.963013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:00.964548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1358/2700] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:00 2022] DELETE /identity/v3/projects/a6b3f80220af4a36b785a8cd7408c923 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:00.969490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:00.981791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa799fd-2e34-46e0-8807-dd7789797002 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:01.001969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:01.043560 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:01.044112 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.044370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.044635 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] PATH_INFO: `/v3/users/76f8f25a0af04661b7517afb00996cec` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.045738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] RBAC: Authorizing `identity:delete_user(user_id=76f8f25a0af04661b7517afb00996cec)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:01.047677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:01.095007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76f8f25a0af04661b7517afb00996cec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.102137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76f8f25a0af04661b7517afb00996cec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.114573 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76f8f25a0af04661b7517afb00996cec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.115282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76f8f25a0af04661b7517afb00996cec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.115896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01b5fe21-da61-43a5-8eae-2de37523e237 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76f8f25a0af04661b7517afb00996cec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.129656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1343/2701] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:00 2022] DELETE /identity/v3/users/76f8f25a0af04661b7517afb00996cec => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:01.433479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:01.434999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:01.449496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce6a1b2-e475-4afa-9f6a-d5106934797f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:01.535590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:01.536225 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.536495 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.536789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] PATH_INFO: `/v3/projects/fa933e312edf4d2f9b42e949c980da54` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.537958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa933e312edf4d2f9b42e949c980da54)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:01.539870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:01.571242 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:01.589929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa933e312edf4d2f9b42e949c980da54', 'request_id': 'req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.597749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa933e312edf4d2f9b42e949c980da54', 'request_id': 'req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.606899 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa933e312edf4d2f9b42e949c980da54', 'request_id': 'req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.617386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] The token cache is being invalidate because project fa933e312edf4d2f9b42e949c980da54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:01.617673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.618587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-45322c52-3da5-4b7d-98a4-b92a66c3e5cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:01.619803 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1359/2702] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:01 2022] DELETE /identity/v3/projects/fa933e312edf4d2f9b42e949c980da54 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:01.624671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.624949 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.625215 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.645316 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:01.646506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:01.724508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1344/2703] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:01.726459 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.726939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.727234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.752780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:01.754078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:01.762911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1360/2704] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:01.767237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:01.784040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a249a-0ddc-4b0a-ba06-251051d2df6c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:01.789860 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:01.790585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.790931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.791261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.791982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:01.792392 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:01.793996 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:01.808378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1345/2705] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:55:01.813024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:01.823656 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19639b-892d-4371-b981-a2f7a61d940f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:01.828493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:01.829096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.829366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.837448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.837448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:01.837448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:01.867747 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1361/2706] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:01 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:01.872073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:01.883933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e6e038-2f7c-496c-84d4-338ea70d980f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:01.911749 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:01.946788 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:01.947246 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:01.947520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:01.948107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:01.948403 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:01.950373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:01.978531 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1346/2707] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:01 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:01.981756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:01.994685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac663efb-1f09-46a4-a64d-0450ccfd38df admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.000658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.001274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.001547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.001817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.002465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.004279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.014683 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1362/2708] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:02.018854 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:02.030072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dcb5c4-c79e-4f21-8f13-888891f62020 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.035377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.035998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.036268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.036565 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.037353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.038927 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fafa2f-7d72-4d81-9607-92f3100ac581 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.050108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1347/2709] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:02.057119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:02.069565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c4c2f-c493-4805-8a5e-42498be64d2e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.074193 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.074897 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.075201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.075435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] PATH_INFO: `/v3/projects/e2c0ce3948424d3590f73b97d5f2808f/users/d36fc5f1d2664b238097cc2105cde5ab/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.091149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2c0ce3948424d3590f73b97d5f2808f, user_id=d36fc5f1d2664b238097cc2105cde5ab, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.093818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.108675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1363/2710] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:02 2022] PUT /identity/v3/projects/e2c0ce3948424d3590f73b97d5f2808f/users/d36fc5f1d2664b238097cc2105cde5ab/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:02.120773 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.121057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.121331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.167352 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] MFA Rules not processed for user `d36fc5f1d2664b238097cc2105cde5ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:02.168119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:02.239488 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1348/2711] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:02.251647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:02.253388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:02.312389 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51468781-83a9-4306-a526-dd9d0359adc5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.316607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.317363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.317632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.317895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.326384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.327876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561e9048-a15d-4ba8-97bf-41040a892799 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.337358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1364/2712] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:02.356506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:02.427024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01091acc-86b7-4ca9-be95-51518363c829 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.431955 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.432580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.432849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.433188 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] PATH_INFO: `/v3/users/d36fc5f1d2664b238097cc2105cde5ab` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.434385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d36fc5f1d2664b238097cc2105cde5ab)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.436497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.481358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:02.504833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd36fc5f1d2664b238097cc2105cde5ab'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:02.513869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd36fc5f1d2664b238097cc2105cde5ab'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:02.530526 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd36fc5f1d2664b238097cc2105cde5ab'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:02.531812 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd36fc5f1d2664b238097cc2105cde5ab'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:02.532845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd36fc5f1d2664b238097cc2105cde5ab'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:02.552239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1349/2713] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:02 2022] DELETE /identity/v3/users/d36fc5f1d2664b238097cc2105cde5ab => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:02.556829 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.557339 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.557450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.581081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:02.583181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:02.641121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:02.656154 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1365/2714] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:02.666527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:02.668424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:02.735836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b6b7bb-75b1-4ea3-9459-22b828cfcae2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:02.740730 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:02.741338 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:02.741606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:02.741871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:02.751178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:02.752924 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cffe05d-2c33-46cb-8418-902a2c98b56f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:02.765107 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1350/2715] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:03.042202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.043752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:03.057624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d1c8c-1ec1-4df1-ad44-fbd545aa70b2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.062182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.062844 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.063117 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.063382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] PATH_INFO: `/v3/projects/e2c0ce3948424d3590f73b97d5f2808f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.064589 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2c0ce3948424d3590f73b97d5f2808f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.066620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.118643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2c0ce3948424d3590f73b97d5f2808f', 'request_id': 'req-e0f0d118-b76a-4bf7-945d-58546eebc37b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:03.124930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2c0ce3948424d3590f73b97d5f2808f', 'request_id': 'req-e0f0d118-b76a-4bf7-945d-58546eebc37b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:03.134532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2c0ce3948424d3590f73b97d5f2808f', 'request_id': 'req-e0f0d118-b76a-4bf7-945d-58546eebc37b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:03.146580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] The token cache is being invalidate because project e2c0ce3948424d3590f73b97d5f2808f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:03.146958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:03.147614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e0f0d118-b76a-4bf7-945d-58546eebc37b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:03.149112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1366/2716] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:03 2022] DELETE /identity/v3/projects/e2c0ce3948424d3590f73b97d5f2808f => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:03.154146 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.154361 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.154641 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.173898 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:03.175158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:03.237713 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1351/2717] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:03.246493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.246818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.247069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.268122 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:03.269605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:03.277618 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1367/2718] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:03.281887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.292375 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8e1f08-632f-4009-9fd3-f7e46745af78 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.297087 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.297700 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.297974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.298241 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.298918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:03.299216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.301435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.315285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1352/2719] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:55:03.327189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.336628 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f82952-dd09-4a40-a3d2-0acfd6cb026b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.342177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.342971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.343314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.343513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.344317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.346169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.377707 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1368/2720] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:03 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:03.382197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.392805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ef73f-cf50-4b30-9cdd-bb1936653410 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.442407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:03.459122 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.459728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.460002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.460269 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.460902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.462882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.489658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1353/2721] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:03 2022] POST /identity/v3/users => generated 289 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:03.494496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.506864 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3ce4a0-c29d-4e6f-b22b-1d2b44a128c8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.511637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.512248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.512523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.512789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.513439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.515070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.526272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1369/2722] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:03.531109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.542262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa62e7b-fdcc-474c-8f70-345b8158ff13 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.547914 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.548528 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.548806 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.549077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.549719 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.551607 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5553eeb4-8c88-41c3-975e-445e2b09d3dc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.562709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1354/2723] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:03.567563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.579181 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00208cc8-640c-4078-8f9b-4cead80f2a1f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.586622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.587295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.587563 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.587838 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] PATH_INFO: `/v3/projects/0dd7cde0d9e84881a89225c5851bee48/users/872e829bdcde470c8ebb235f7279ed6f/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.603913 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0dd7cde0d9e84881a89225c5851bee48, user_id=872e829bdcde470c8ebb235f7279ed6f, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.606854 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.620749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1370/2724] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:03 2022] PUT /identity/v3/projects/0dd7cde0d9e84881a89225c5851bee48/users/872e829bdcde470c8ebb235f7279ed6f/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:03.690513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.690917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.691341 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.721865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] MFA Rules not processed for user `872e829bdcde470c8ebb235f7279ed6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:03.723981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:03.808070 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1355/2725] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:03.820686 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:03.822481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:03.893189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386cbb54-cd39-4c5d-9d8f-aed1c975d79b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:03.898368 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:03.899458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:03.899458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:03.899820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:03.912129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:03.913600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fdbd61-375d-489c-81a1-89328ca91c80 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:03.926970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1371/2726] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:12.087714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:12.089851 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.166780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e48ea4d-b972-4310-a584-699685ca8a33 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:12.171740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:12.172364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:12.172648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:12.172902 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] PATH_INFO: `/v3/users/683fa5fa5ed14c4f9244817664e3c8be` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:12.174012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=683fa5fa5ed14c4f9244817664e3c8be)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:12.176419 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:12.229303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '683fa5fa5ed14c4f9244817664e3c8be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.236923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '683fa5fa5ed14c4f9244817664e3c8be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.238024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:12.252483 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '683fa5fa5ed14c4f9244817664e3c8be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.253954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '683fa5fa5ed14c4f9244817664e3c8be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.254780 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '683fa5fa5ed14c4f9244817664e3c8be'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.274036 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1356/2727] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:12 2022] DELETE /identity/v3/users/683fa5fa5ed14c4f9244817664e3c8be => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:12.279144 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:12.279667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:12.279667 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:12.299970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:12.301171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.316359 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:12.370201 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1372/2728] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:12.381235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:12.382706 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.463757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff9210-f50d-48ea-9f95-2444718b0a32 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:12.467716 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:12.468474 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:12.468609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:12.468862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:12.479649 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:12.482727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:12.492387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1357/2729] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:12.535649 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:12.537132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.550284 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf39133-9210-45b5-867a-d361b81ad1f2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:12.555155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:12.555774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:12.556163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:12.556631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:12.560371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.579917 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:12.581852 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:12.593473 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1373/2730] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:12.836364 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:12.838145 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:12.859092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991d6b66-e6f6-487b-88c6-747c5d2e56ea service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:12.867656 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:12.868032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:12.868406 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:12.868598 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] PATH_INFO: `/v3/projects/afdcf85bf60b4105a07b7e04123f94e8` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:12.869779 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] RBAC: Authorizing `identity:delete_project(project_id=afdcf85bf60b4105a07b7e04123f94e8)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:12.872543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:12.937060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afdcf85bf60b4105a07b7e04123f94e8', 'request_id': 'req-d4dc7ad1-904f-4007-ae74-f569089b9020'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.948907 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afdcf85bf60b4105a07b7e04123f94e8', 'request_id': 'req-d4dc7ad1-904f-4007-ae74-f569089b9020'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.960821 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afdcf85bf60b4105a07b7e04123f94e8', 'request_id': 'req-d4dc7ad1-904f-4007-ae74-f569089b9020'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.976253 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] The token cache is being invalidate because project afdcf85bf60b4105a07b7e04123f94e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:12.976722 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.981952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:12.981952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1358/2731] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:12 2022] DELETE /identity/v3/projects/afdcf85bf60b4105a07b7e04123f94e8 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:16.374057 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:16.375616 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:16.395133 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0763cd-95cc-44e9-9e3f-ed34a2bf4250 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:16.466116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:16.466796 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:16.467100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:16.467377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] PATH_INFO: `/v3/users/78634b34107d49cdbf0126014bf8f6f5` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:16.468532 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] RBAC: Authorizing `identity:delete_user(user_id=78634b34107d49cdbf0126014bf8f6f5)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:16.470469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:16.535170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78634b34107d49cdbf0126014bf8f6f5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.543491 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78634b34107d49cdbf0126014bf8f6f5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.556073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78634b34107d49cdbf0126014bf8f6f5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.556733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78634b34107d49cdbf0126014bf8f6f5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.564089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d598e91e-fada-4431-a9dd-74fa6a79756f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78634b34107d49cdbf0126014bf8f6f5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.571776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1374/2732] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:16 2022] DELETE /identity/v3/users/78634b34107d49cdbf0126014bf8f6f5 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:16.788813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:16.790785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:16.809831 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc7ad1-904f-4007-ae74-f569089b9020 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:16.890313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:16.891458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:16.891874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:16.892072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] PATH_INFO: `/v3/projects/5d21789971964770bdf417547cd19840` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:16.894038 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d21789971964770bdf417547cd19840)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:16.896172 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:16.898155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:16.959679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d21789971964770bdf417547cd19840', 'request_id': 'req-4f95036b-7ed2-4444-8466-cc5be42f8dee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.966622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d21789971964770bdf417547cd19840', 'request_id': 'req-4f95036b-7ed2-4444-8466-cc5be42f8dee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:16.992323 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d21789971964770bdf417547cd19840', 'request_id': 'req-4f95036b-7ed2-4444-8466-cc5be42f8dee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:17.003905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] The token cache is being invalidate because project 5d21789971964770bdf417547cd19840 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:17.004184 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:17.004840 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4f95036b-7ed2-4444-8466-cc5be42f8dee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:17.006355 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1359/2733] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:16 2022] DELETE /identity/v3/projects/5d21789971964770bdf417547cd19840 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:17.011376 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b565e-e201-4834-8413-e3165f907dea None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.011661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b565e-e201-4834-8413-e3165f907dea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.011935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11b565e-e201-4834-8413-e3165f907dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.033451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d11b565e-e201-4834-8413-e3165f907dea None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:17.035352 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d11b565e-e201-4834-8413-e3165f907dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:17.053468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:17.101774 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1375/2734] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:17.112732 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.113129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.113129 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.146059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:17.146059 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:17.158743 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1360/2735] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:17.161680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b565e-e201-4834-8413-e3165f907dea None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.173884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11b565e-e201-4834-8413-e3165f907dea None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.181583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.182663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.183140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.183619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.184658 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:17.185163 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.187134 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.204272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1376/2736] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:55:17.209140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.219199 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e504268-2af2-48c0-b5a3-4c6e1cc1dd7c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.226794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.227824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.228329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.228823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.230033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.233481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.277986 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1361/2737] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/projects => generated 379 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:17.285976 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.296290 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13916ac8-6cae-4fa5-8a5b-f690f383d333 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.358511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.359320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.359498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.360016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.360299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.363433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.386493 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1377/2738] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:17.396785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.409883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b8d2-63b4-498a-8d92-cbd82ebac688 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.415799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.416536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.416833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.417137 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.417816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.419480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.433823 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1362/2739] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:17.438636 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.450291 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0259996-2377-42ec-bcdc-231190d4a06c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.455544 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.456587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.456734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.457070 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.457786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.460088 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e09ded-47da-4990-a512-ab057f639539 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.474498 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1378/2740] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:17 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:17.479011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.490615 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45482793-4457-40b6-b231-6210e6305343 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.495981 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.496982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.497255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.497527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] PATH_INFO: `/v3/projects/283af8631c474dcb942caca02e7bda85/users/0cda0f054b6a4de0bf63c7fb43fd6083/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.517464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=283af8631c474dcb942caca02e7bda85, user_id=0cda0f054b6a4de0bf63c7fb43fd6083, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.520261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.533830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1363/2741] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:17 2022] PUT /identity/v3/projects/283af8631c474dcb942caca02e7bda85/users/0cda0f054b6a4de0bf63c7fb43fd6083/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:17.546232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.546512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.546885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.579929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] MFA Rules not processed for user `0cda0f054b6a4de0bf63c7fb43fd6083`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:17.581422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:17.649694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1379/2742] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:17.654442 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.727260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc32d2e7-c015-46d3-9bdf-37e1a1f4afcb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.732133 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.732747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.733020 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.733286 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.733933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.735869 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.766022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1364/2743] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:17.772345 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.783955 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e1c643-7d81-4623-b6d3-9a9a257521ce None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.797834 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:17.852409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.853155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.853547 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.853911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.855253 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.857354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.884716 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1380/2744] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:17 2022] POST /identity/v3/users => generated 296 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:17.888880 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.900508 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0935b8e2-2c51-4150-8c61-2473e8fe29b3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.905402 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.906003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.906274 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.906536 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.907197 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.908847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.915242 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:17.923002 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1365/2745] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:17 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:17.928407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:17.940918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2283a7-8c1f-4164-97cf-421f4e280df1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:17.947756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:17.948798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:17.949120 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:17.949487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] PATH_INFO: `/v3/projects/f7daa162d98941c191e26a6836f0f191/users/6a58c106273446298cf27f265354487f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:17.968484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7daa162d98941c191e26a6836f0f191, user_id=6a58c106273446298cf27f265354487f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:17.973440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:17.988301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1381/2746] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:17 2022] PUT /identity/v3/projects/f7daa162d98941c191e26a6836f0f191/users/6a58c106273446298cf27f265354487f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:17.993571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:18.010189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45af6dfd-90ad-424d-ada9-60bc1c05eeff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:18.078024 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:18.078671 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:18.078947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:18.079235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:18.079939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:18.082459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d2761-643e-4479-ba40-2c8b3cdd4155 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:18.098300 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1366/2747] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:17 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:18.103874 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:18.115216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd6cf70-0daa-4eeb-bea0-26ebe7b54274 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:18.120058 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:18.120727 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:18.121004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:18.121278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] PATH_INFO: `/v3/projects/f7daa162d98941c191e26a6836f0f191/users/6a58c106273446298cf27f265354487f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:18.123388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7daa162d98941c191e26a6836f0f191, user_id=6a58c106273446298cf27f265354487f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:18.126169 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:18.141324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1382/2748] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:18 2022] PUT /identity/v3/projects/f7daa162d98941c191e26a6836f0f191/users/6a58c106273446298cf27f265354487f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:18.157441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:18.157441 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:18.157657 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:18.189490 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] MFA Rules not processed for user `6a58c106273446298cf27f265354487f`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:18.191497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:18.267699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1367/2749] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:18 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:18.279851 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:18.281574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:18.354056 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1b6b88-73ad-451c-b1a5-5c13a2abed1d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:18.357822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:18.358441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:18.358748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:18.359020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:18.370270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:18.372038 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:18.383512 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1383/2750] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:18 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:18.481076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:18.482529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:18.500756 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc7c09f-640a-45d0-978d-b9ab18728c67 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:18.505332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:18.505950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:18.506227 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:18.506500 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:18.567399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:18.568833 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d703f99-a558-40ce-a5b0-0043d52a37df service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:18.577959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1368/2751] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:18 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:25.246347 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:25.248023 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:25.320881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec2beb-1436-45ae-970e-d9fa23bb9414 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:25.326826 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:25.327468 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:25.327741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:25.328016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] PATH_INFO: `/v3/users/872e829bdcde470c8ebb235f7279ed6f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:25.329417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] RBAC: Authorizing `identity:delete_user(user_id=872e829bdcde470c8ebb235f7279ed6f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:25.332517 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:25.349530 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:25.403849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '872e829bdcde470c8ebb235f7279ed6f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:25.412296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '872e829bdcde470c8ebb235f7279ed6f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:25.425178 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '872e829bdcde470c8ebb235f7279ed6f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:25.425752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '872e829bdcde470c8ebb235f7279ed6f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:25.436420 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '872e829bdcde470c8ebb235f7279ed6f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:25.445748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1384/2752] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:25 2022] DELETE /identity/v3/users/872e829bdcde470c8ebb235f7279ed6f => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:25.452164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:25.452676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:25.453134 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:25.487329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:25.488931 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:25.503317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:25.558969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1369/2753] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:25.570136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:25.571621 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:25.637996 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caced3d-011b-4c01-a56b-d07ceb724feb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:25.641758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:25.642371 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:25.642687 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:25.643020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:25.654279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:25.655869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4392d574-1291-48bc-8cb0-99537e7b4b10 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:25.666511 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1385/2754] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:25.987832 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:25.989836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:26.005757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cd3e39-2432-490c-911b-fddd7a8949e5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.011876 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.012563 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.021459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.021998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] PATH_INFO: `/v3/projects/0dd7cde0d9e84881a89225c5851bee48` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.023570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0dd7cde0d9e84881a89225c5851bee48)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.025620 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.091961 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0dd7cde0d9e84881a89225c5851bee48', 'request_id': 'req-edc91639-9462-4792-88fa-168eac5b3277'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.099715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0dd7cde0d9e84881a89225c5851bee48', 'request_id': 'req-edc91639-9462-4792-88fa-168eac5b3277'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.110167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0dd7cde0d9e84881a89225c5851bee48', 'request_id': 'req-edc91639-9462-4792-88fa-168eac5b3277'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.123655 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] The token cache is being invalidate because project 0dd7cde0d9e84881a89225c5851bee48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:26.123941 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.124644 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-edc91639-9462-4792-88fa-168eac5b3277 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.126342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1370/2755] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:25 2022] DELETE /identity/v3/projects/0dd7cde0d9e84881a89225c5851bee48 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:26.133083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.133542 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.134000 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.161280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:26.162507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:26.225286 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1386/2756] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:26.239141 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.239425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.239593 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.261681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:26.262964 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:26.272529 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1371/2757] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:26.277961 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.296156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47edecf-f76e-4f5e-8c90-ab2c65ae4837 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.303905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.304949 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.305845 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.306361 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.307659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:26.308200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.311849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.331740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1387/2758] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:55:26.338416 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.348522 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d7f0ed-a48a-4ff1-9c4a-9dde69794144 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.356867 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.357493 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.357772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.358037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.359739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.362031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.395424 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1372/2759] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:26.399841 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.410752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243477c3-9321-42b1-9291-46a1032fe0b0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.437494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:26.481423 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.482044 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.482336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.482610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.483332 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.485272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.509685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1388/2760] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/users => generated 288 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:26.514182 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.526583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afa324c-5e32-470a-8733-8d9ae5848744 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.532360 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.533081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.533353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.534016 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.534736 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.536372 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.552138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1373/2761] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:26.557424 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.571560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926d471-d04e-4294-81ab-074a050e2522 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.578467 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.579446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.579877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.580330 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.581937 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.584609 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5464f9b-6d7f-4a06-bad1-1ee53ced919e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.601666 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1389/2762] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:26 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:26.606328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.621954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d96668-ad10-4d48-a5f0-991afffba18a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.630886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.631890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.632362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.632791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] PATH_INFO: `/v3/projects/9058c11346934a569c3d0dcb74655b6b/users/e5822e08d63c4181a2ffa4cfcf0fb8ce/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.660394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=9058c11346934a569c3d0dcb74655b6b, user_id=e5822e08d63c4181a2ffa4cfcf0fb8ce, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.663190 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.683100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1374/2763] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:26 2022] PUT /identity/v3/projects/9058c11346934a569c3d0dcb74655b6b/users/e5822e08d63c4181a2ffa4cfcf0fb8ce/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:26.694206 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.694676 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.695083 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.733304 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] MFA Rules not processed for user `e5822e08d63c4181a2ffa4cfcf0fb8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:26.734541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:26.775846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.777606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:26.812769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1390/2764] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:26.817912 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.829591 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:26.868260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc98ba9-59a4-4c10-b1da-330b8a3441fe admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.873430 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.874081 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.874353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.874653 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] PATH_INFO: `/v3/users/0cda0f054b6a4de0bf63c7fb43fd6083` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.875891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cda0f054b6a4de0bf63c7fb43fd6083)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.878447 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.897720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e1a819-bc02-46f9-b392-6efb4641fcb5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:26.903004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:26.903614 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:26.903890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:26.904150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:26.904792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:26.906904 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:26.941883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cda0f054b6a4de0bf63c7fb43fd6083'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.947216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1391/2765] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/projects => generated 375 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:26.953056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cda0f054b6a4de0bf63c7fb43fd6083'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.954215 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:26.977202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cda0f054b6a4de0bf63c7fb43fd6083'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.978209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cda0f054b6a4de0bf63c7fb43fd6083'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.979218 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c971a106-0507-4d44-a665-c7345c76151e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cda0f054b6a4de0bf63c7fb43fd6083'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:26.982377 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:26.997947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1375/2766] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:26 2022] DELETE /identity/v3/users/0cda0f054b6a4de0bf63c7fb43fd6083 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:27.004643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.005118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.005484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.029973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:27.032643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.053277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86532352-ce95-4e0a-bb6c-840fd6d34d23 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.116634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1376/2767] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:27.126676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.128977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.133412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.134034 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.134482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.134798 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.135480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.137661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.166639 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1392/2768] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:26 2022] POST /identity/v3/users => generated 294 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:27.170815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.173379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.194212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167ae355-ffd0-48a3-a11e-a86b2484751c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.200349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.200974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.201248 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.201516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.202409 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.205094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.209161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612c02fd-812b-4127-80c5-d583e8b4e9ab None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.214367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.215231 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.215597 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.216083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.218406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1393/2769] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:27 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:27.223611 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.228917 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.230248 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.234103 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976c645b-36f7-4c5e-a165-4ee9b6d7d071 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.240780 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.241479 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.241759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.242030 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] PATH_INFO: `/v3/projects/90c42908fdd04dcebe232717f2c214f3/users/fab4adc02e5049588b31d823310f9581/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.242953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1377/2770] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:27.260731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=90c42908fdd04dcebe232717f2c214f3, user_id=fab4adc02e5049588b31d823310f9581, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.263492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.279603 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1394/2771] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:27 2022] PUT /identity/v3/projects/90c42908fdd04dcebe232717f2c214f3/users/fab4adc02e5049588b31d823310f9581/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:27.284365 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.289332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:27.295326 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e412037-33bb-423b-b0ce-5a2599469c6b service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.390013 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.390707 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.390988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.391238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.391937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.393735 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2562e195-146c-43f6-9ce2-340097fa4ac1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.414619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1378/2772] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:27 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:27.416548 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.430340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780881fb-d695-4e78-91b7-0f8d1cb2ef8e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.435970 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.436790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.437155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.437513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] PATH_INFO: `/v3/projects/90c42908fdd04dcebe232717f2c214f3/users/fab4adc02e5049588b31d823310f9581/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.439720 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90c42908fdd04dcebe232717f2c214f3, user_id=fab4adc02e5049588b31d823310f9581, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.442678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.454456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1395/2773] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:27 2022] PUT /identity/v3/projects/90c42908fdd04dcebe232717f2c214f3/users/fab4adc02e5049588b31d823310f9581/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:27.465728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.466882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.466882 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.507200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] MFA Rules not processed for user `fab4adc02e5049588b31d823310f9581`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:27.508524 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.605119 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1379/2774] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:27 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:27.621931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.623739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.653820 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.656207 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.740267 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fbb871-1461-4cfc-ad3f-70eeee97cc06 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.746179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.748085 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.748418 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.748648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.750914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:27.773349 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:27.789157 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f062adfb-6c55-4cd9-9a44-c5127d717186 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:27.797445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:27.799303 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:27.799631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:27.800075 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] PATH_INFO: `/v3/projects/283af8631c474dcb942caca02e7bda85` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:27.802627 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=283af8631c474dcb942caca02e7bda85)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.804901 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.826308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:27.827863 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:27.843503 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1396/2775] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:27.846837 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:27.871870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '283af8631c474dcb942caca02e7bda85', 'request_id': 'req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:27.880682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '283af8631c474dcb942caca02e7bda85', 'request_id': 'req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:27.891772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '283af8631c474dcb942caca02e7bda85', 'request_id': 'req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:27.911110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] The token cache is being invalidate because project 283af8631c474dcb942caca02e7bda85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:27.911582 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:27.912436 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:27.919503 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1380/2776] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:27 2022] DELETE /identity/v3/projects/283af8631c474dcb942caca02e7bda85 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:27.922040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:27.936256 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb244b2-a4fa-4f7e-ae50-c2aa74f2b044 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:28.018754 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:28.019408 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.019680 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.019952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] PATH_INFO: `/v3/users/6a58c106273446298cf27f265354487f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.021135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a58c106273446298cf27f265354487f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:28.023285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:28.081637 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a58c106273446298cf27f265354487f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.091938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a58c106273446298cf27f265354487f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.110478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a58c106273446298cf27f265354487f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.111558 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a58c106273446298cf27f265354487f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.122965 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15e5d87a-8bdc-49d9-ac9d-a235257e414f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a58c106273446298cf27f265354487f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.135593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1397/2777] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:27 2022] DELETE /identity/v3/users/6a58c106273446298cf27f265354487f => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:28.483427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:28.484847 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:28.500849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a198b2-f4f4-44cb-b4f8-d921c604e6d4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:28.560600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:28.561230 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.561507 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.561777 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] PATH_INFO: `/v3/projects/f7daa162d98941c191e26a6836f0f191` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.562932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7daa162d98941c191e26a6836f0f191)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:28.565000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:28.614407 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7daa162d98941c191e26a6836f0f191', 'request_id': 'req-38b8b607-141f-47a3-a8a2-66beedfa7b7e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.620579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7daa162d98941c191e26a6836f0f191', 'request_id': 'req-38b8b607-141f-47a3-a8a2-66beedfa7b7e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.627080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7daa162d98941c191e26a6836f0f191', 'request_id': 'req-38b8b607-141f-47a3-a8a2-66beedfa7b7e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.638237 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] The token cache is being invalidate because project f7daa162d98941c191e26a6836f0f191 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:28.638511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.639163 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38b8b607-141f-47a3-a8a2-66beedfa7b7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:28.640715 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1381/2778] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:28 2022] DELETE /identity/v3/projects/f7daa162d98941c191e26a6836f0f191 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:28.645617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.646118 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.646247 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.666135 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:28.667374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:28.700009 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:28.728428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1398/2779] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:28.736077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.736354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.736624 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.762084 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:28.763328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:28.772366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1382/2780] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:28.777041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:28.787869 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041b0e36-ca8b-4e06-bddd-267a1b9788c9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:28.792675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:28.793301 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.793581 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.793850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.794510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:28.794853 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:28.796550 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:28.810804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1399/2781] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:55:28.815367 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:28.824843 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520efe36-10f5-45ac-9c06-eda1ab544d18 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:28.829668 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:28.830285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.830745 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.831014 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.831740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:28.833638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:28.864290 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1383/2782] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:28 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:28.869061 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:28.879693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca37e6-6a98-44e3-8af3-a60f9d39bf91 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:28.952322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:28.952936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:28.953216 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:28.953484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:28.954159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:28.956756 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:28.983176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1400/2783] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:28 2022] POST /identity/v3/users => generated 282 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:28.984480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:28.995727 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22a026a-abb9-4baf-99f8-1262c2d4d2aa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.000394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.000998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.001273 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.001547 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.002205 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.004605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.016285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1384/2784] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:29.020708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.030647 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7639b20b-8777-420a-8d9e-3d69cfb8a09e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.035346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.035935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.036211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.036478 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.037112 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.038714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dff685-22e7-46cb-b172-4a90f3522231 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.050924 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1401/2785] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:29.055351 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.062019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:29.066842 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a01ee3-21b1-465b-bb60-79f0fe2552e8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.072993 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.073805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.074093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.074738 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] PATH_INFO: `/v3/projects/ccd982fbdc474ae784c29d623e5271b8/users/e554eaca815b469986b08da880530935/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.091057 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccd982fbdc474ae784c29d623e5271b8, user_id=e554eaca815b469986b08da880530935, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.093699 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.107160 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1385/2786] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:29 2022] PUT /identity/v3/projects/ccd982fbdc474ae784c29d623e5271b8/users/e554eaca815b469986b08da880530935/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:29.119171 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.119460 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.119739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.151398 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] MFA Rules not processed for user `e554eaca815b469986b08da880530935`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:29.152884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:29.229750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1402/2787] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:29 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:29.234918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.315001 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a15d218-5d92-4243-a638-09815ee94482 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.320054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.320794 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.321003 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.321187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.321818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.323834 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.371327 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1386/2788] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:29 2022] POST /identity/v3/projects => generated 365 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:29.375374 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.387876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a66b1a6-781d-4b3f-b76f-6362f9e3b741 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.425249 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:29.468715 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.469363 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.469638 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.469906 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.470657 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.473186 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.497555 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1403/2789] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:29 2022] POST /identity/v3/users => generated 289 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:29.501785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.513543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592e9a-88d8-4fe9-86c1-dfaa3ac93c56 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.518264 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.518894 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.519167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.519433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.520117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.521747 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.532953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1387/2790] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:29.538981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.550205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ad1bb-c24a-49f0-a369-5739d2a88247 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.556152 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.556818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.557145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.557443 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] PATH_INFO: `/v3/projects/f79924166ec14605b4647933fb66b164/users/53623f1bc7094950afcf14aee5949aa9/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.574782 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f79924166ec14605b4647933fb66b164, user_id=53623f1bc7094950afcf14aee5949aa9, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.577629 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.592823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1404/2791] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:29 2022] PUT /identity/v3/projects/f79924166ec14605b4647933fb66b164/users/53623f1bc7094950afcf14aee5949aa9/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:29.598366 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.609050 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1025e0e5-9ef5-46dc-a815-0aae68d50f7b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.673726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.675181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.675181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.675434 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.676077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.677652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af450e8d-6326-401a-bf31-1e31b1965384 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.688786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1388/2792] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:29 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:29.693218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.703496 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fae79cb-4261-4cd9-bce4-cf57b860b171 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.708971 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.709612 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.709890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.710160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] PATH_INFO: `/v3/projects/f79924166ec14605b4647933fb66b164/users/53623f1bc7094950afcf14aee5949aa9/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.712108 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f79924166ec14605b4647933fb66b164, user_id=53623f1bc7094950afcf14aee5949aa9, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.714790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.726017 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1405/2793] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:29 2022] PUT /identity/v3/projects/f79924166ec14605b4647933fb66b164/users/53623f1bc7094950afcf14aee5949aa9/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:29.737645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.737928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.738202 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.770073 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] MFA Rules not processed for user `53623f1bc7094950afcf14aee5949aa9`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:29.771383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:29.812131 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:29.841665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1389/2794] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:29 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:29.853988 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:29.855878 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:29.916024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c40b304-0189-4f5b-8d5f-8579f3f4d961 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:29.919623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:29.920224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:29.920497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:29.920764 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:29.930474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:29.932317 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:29.942446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1406/2795] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:29 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:30.397776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:30.463288 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210c5b3-7de7-45ec-b04b-6f9f154270a3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:30.468214 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:30.468886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.469117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.469394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] PATH_INFO: `/v3/users/e554eaca815b469986b08da880530935` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.470574 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e554eaca815b469986b08da880530935)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:30.472588 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:30.522477 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:30.523180 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e554eaca815b469986b08da880530935'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.523877 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:30.530572 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e554eaca815b469986b08da880530935'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.539458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dcc61e-7a77-4b10-9aab-e9b60ba28b33 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:30.543605 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e554eaca815b469986b08da880530935'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.543932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:30.544426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e554eaca815b469986b08da880530935'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.544541 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.544804 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.545047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b0e3c52-c03a-4a4b-b31c-9310cd88a66c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e554eaca815b469986b08da880530935'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.545156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] PATH_INFO: `/v3/users/e5822e08d63c4181a2ffa4cfcf0fb8ce` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.546161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5822e08d63c4181a2ffa4cfcf0fb8ce)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:30.548602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:30.559094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1390/2796] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:30 2022] DELETE /identity/v3/users/e554eaca815b469986b08da880530935 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:30.563654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.563987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.564258 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.586610 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:30.587824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:30.605373 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5822e08d63c4181a2ffa4cfcf0fb8ce'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.613763 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5822e08d63c4181a2ffa4cfcf0fb8ce'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.626432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5822e08d63c4181a2ffa4cfcf0fb8ce'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.628633 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5822e08d63c4181a2ffa4cfcf0fb8ce'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.636832 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-21350b26-7dee-482f-8b56-fdafacbc4b42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5822e08d63c4181a2ffa4cfcf0fb8ce'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:30.639432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:INSERT=6,keystone:DELETE=10 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:30.644551 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1407/2797] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:30 2022] DELETE /identity/v3/users/e5822e08d63c4181a2ffa4cfcf0fb8ce => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:30.649488 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.649977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.650278 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.656332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:30.670693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1391/2798] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:30.674653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:30.676067 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:30.681268 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:30.682659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:30.748703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df1dde2-5c24-4649-b130-a0f35b67ec85 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:30.754308 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1408/2799] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:30.764258 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:30.765670 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:30.803144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:30.803770 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.804048 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.804321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.814108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:30.815606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:30.824124 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b0303f-cd70-4750-bc30-0da45ff7ed5d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:30.825213 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1392/2800] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:30.828129 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:30.828817 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:30.829166 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:30.829506 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:30.839746 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:30.841346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:30.852050 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1409/2801] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:31.220032 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.221666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:31.236091 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f8072f-3455-4acb-b71f-0b4547066a21 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:31.243766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.245815 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:31.246383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:31.247285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:31.247634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:31.248033 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] PATH_INFO: `/v3/projects/ccd982fbdc474ae784c29d623e5271b8` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:31.249807 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccd982fbdc474ae784c29d623e5271b8)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:31.253167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:31.270092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3c42e-8fbc-47aa-85a4-de5ab387e257 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:31.283434 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:31.284320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:31.284445 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:31.284673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] PATH_INFO: `/v3/projects/9058c11346934a569c3d0dcb74655b6b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:31.286008 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9058c11346934a569c3d0dcb74655b6b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:31.287895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:31.345169 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccd982fbdc474ae784c29d623e5271b8', 'request_id': 'req-8425d8a6-4ae3-48b4-994a-0b88142823c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.352624 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9058c11346934a569c3d0dcb74655b6b', 'request_id': 'req-bc8cfa88-9308-4e21-a419-c45fa9b4b693'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.352912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccd982fbdc474ae784c29d623e5271b8', 'request_id': 'req-8425d8a6-4ae3-48b4-994a-0b88142823c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.362680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccd982fbdc474ae784c29d623e5271b8', 'request_id': 'req-8425d8a6-4ae3-48b4-994a-0b88142823c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.363022 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9058c11346934a569c3d0dcb74655b6b', 'request_id': 'req-bc8cfa88-9308-4e21-a419-c45fa9b4b693'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.373069 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9058c11346934a569c3d0dcb74655b6b', 'request_id': 'req-bc8cfa88-9308-4e21-a419-c45fa9b4b693'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.375977 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] The token cache is being invalidate because project ccd982fbdc474ae784c29d623e5271b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:31.376265 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.376913 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.379486 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1393/2802] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/projects/ccd982fbdc474ae784c29d623e5271b8 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:31.383947 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.385757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:31.390173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] The token cache is being invalidate because project 9058c11346934a569c3d0dcb74655b6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:31.390469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.391160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.393554 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1410/2803] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/projects/9058c11346934a569c3d0dcb74655b6b => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:31.399476 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.400990 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:31.436823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:31.458394 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:31.482448 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cfa88-9308-4e21-a419-c45fa9b4b693 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:31.485858 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8425d8a6-4ae3-48b4-994a-0b88142823c4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:31.489001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:31.489001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:31.489001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:31.489416 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] PATH_INFO: `/v3/users/fab4adc02e5049588b31d823310f9581` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:31.490451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fab4adc02e5049588b31d823310f9581)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:31.491632 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:31.492262 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:31.492518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:31.492652 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:31.492814 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] PATH_INFO: `/v3/users/53623f1bc7094950afcf14aee5949aa9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:31.494118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] RBAC: Authorizing `identity:delete_user(user_id=53623f1bc7094950afcf14aee5949aa9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:31.496342 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:31.547426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fab4adc02e5049588b31d823310f9581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.548618 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53623f1bc7094950afcf14aee5949aa9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.557031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fab4adc02e5049588b31d823310f9581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.557992 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53623f1bc7094950afcf14aee5949aa9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.581676 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53623f1bc7094950afcf14aee5949aa9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.582479 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53623f1bc7094950afcf14aee5949aa9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.583294 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53623f1bc7094950afcf14aee5949aa9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.585402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fab4adc02e5049588b31d823310f9581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.586341 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fab4adc02e5049588b31d823310f9581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.595958 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fab4adc02e5049588b31d823310f9581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:31.600925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1394/2804] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/users/53623f1bc7094950afcf14aee5949aa9 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:31.603735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1411/2805] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/users/fab4adc02e5049588b31d823310f9581 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:31.928850 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.930440 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:31.933138 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:31.935952 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.026759 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae141be-afa4-4603-92e6-af2269a6b58b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.030975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f0cb7d-177c-439a-97da-56fd6f687ed3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.031714 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.032445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.032799 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.033080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] PATH_INFO: `/v3/projects/90c42908fdd04dcebe232717f2c214f3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.035285 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90c42908fdd04dcebe232717f2c214f3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.037016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.037334 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.037669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.037945 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.038223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] PATH_INFO: `/v3/projects/f79924166ec14605b4647933fb66b164` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.040285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=f79924166ec14605b4647933fb66b164)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.044200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.100835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90c42908fdd04dcebe232717f2c214f3', 'request_id': 'req-92e91e24-13d3-46f5-a9de-94ac9811aca6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.111975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90c42908fdd04dcebe232717f2c214f3', 'request_id': 'req-92e91e24-13d3-46f5-a9de-94ac9811aca6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.118377 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:32.123910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:32.126116 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f79924166ec14605b4647933fb66b164', 'request_id': 'req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.128709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90c42908fdd04dcebe232717f2c214f3', 'request_id': 'req-92e91e24-13d3-46f5-a9de-94ac9811aca6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.138867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f79924166ec14605b4647933fb66b164', 'request_id': 'req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.154312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] The token cache is being invalidate because project 90c42908fdd04dcebe232717f2c214f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:32.154312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.154312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-92e91e24-13d3-46f5-a9de-94ac9811aca6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.154312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1395/2806] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/projects/90c42908fdd04dcebe232717f2c214f3 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:32.156939 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f79924166ec14605b4647933fb66b164', 'request_id': 'req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.162781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99ef80-4282-4b27-aeb0-fd2e6b16b545 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.163062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99ef80-4282-4b27-aeb0-fd2e6b16b545 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.163335 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99ef80-4282-4b27-aeb0-fd2e6b16b545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.171353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] The token cache is being invalidate because project f79924166ec14605b4647933fb66b164 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:32.171818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.172824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-1c29318c-4b9d-4dc1-bc08-3ae811cf36bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:32.175305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1412/2807] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:31 2022] DELETE /identity/v3/projects/f79924166ec14605b4647933fb66b164 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:32.180271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4dd8d3-1c8c-4b8e-9f45-69a2e782de8f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.180625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4dd8d3-1c8c-4b8e-9f45-69a2e782de8f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.180827 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4dd8d3-1c8c-4b8e-9f45-69a2e782de8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.185516 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ba99ef80-4282-4b27-aeb0-fd2e6b16b545 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.186871 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ba99ef80-4282-4b27-aeb0-fd2e6b16b545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.204090 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3e4dd8d3-1c8c-4b8e-9f45-69a2e782de8f None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.205909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3e4dd8d3-1c8c-4b8e-9f45-69a2e782de8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.263585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1396/2808] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:32.272271 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.272541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.272818 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.292499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1413/2809] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:32.296343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.297679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.300734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.301027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.301365 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.308093 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1397/2810] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:32.312680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.323113 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de839df-9dec-4303-816a-e07d2abf7ef7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.325685 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.326963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.328302 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.328740 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.328989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.329260 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.329918 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:32.330370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.332178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.337552 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1414/2811] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:32.342465 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.345580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1398/2812] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:55:32.351643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.354157 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac403e7-3dbe-48df-b9d9-d99b156ca2ae None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.359238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.359859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.360622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.360622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.362438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:32.362438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.363868 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cd1f0d-e26a-4aec-9323-9d69d18d5cc7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.364932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.370813 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.371435 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.371705 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.371968 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.373201 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.375733 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.378535 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1415/2813] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 04 08:55:32.383080 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.393932 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd38cd4-4664-4a70-a941-2bb399d1cf76 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.400211 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.401296 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.401641 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.402412 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.404830 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.407195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.415728 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1399/2814] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:32.421186 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.451254 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1416/2815] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/projects => generated 401 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:32.456096 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.502638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b76e4-861b-4a9a-9bfc-2b87d282c92f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.537946 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6f2e13-8421-4d91-980a-bded3832ef6d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.543590 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.544235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.544513 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.544783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.545428 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.547439 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.568816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:32.573839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1417/2816] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/users => generated 301 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:32.578280 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.584648 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.585368 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.585645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.585928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.586602 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.588589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.592300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae91e6-c352-4d0e-bd0e-0ef8beaeaeb2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.598510 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.599695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.600354 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.600632 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.601922 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.603957 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.617540 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1400/2817] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:32.621831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1418/2818] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:32.622805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.622964 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.634076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed3baaf-1b11-4ae5-9275-8f7f5af93279 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.634360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44774a1e-5e95-4864-8483-8c9cfdf0f043 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.639019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.639629 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.639761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.639989 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.640255 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.640369 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.640648 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.640915 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.641041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.641604 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.642539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.644092 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.655185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1419/2819] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:32.656015 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1401/2820] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:32.661080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.661366 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.668130 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:32.671539 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10329c2c-d1f6-4eef-8e69-0d6e0bd38fa8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.676735 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6c681e-b8d8-4bb0-88a3-16d0ba0bd18a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.677890 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.678855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.679679 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.680144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.681276 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.683689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.683939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.684294 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.684947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.685222 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] PATH_INFO: `/v3/projects/f890a618ebb546569fc8db843cce58e4/users/801739c66bbd4ca4b0fac1215c736a30/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.701775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1402/2821] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:32.706804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.709384 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f890a618ebb546569fc8db843cce58e4, user_id=801739c66bbd4ca4b0fac1215c736a30, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.712766 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d2ef45-0fd6-4d77-847f-62cf4e8d0d8a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.717844 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1382576d-283f-4697-b8a9-f80f22e5a67d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.723944 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.724647 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.724925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.725194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] PATH_INFO: `/v3/projects/8a08fa28d7374a029062132a73bb8ab6/users/b62e31d5979c426fb5b6416f087692ff/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.729231 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1420/2822] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:32 2022] PUT /identity/v3/projects/f890a618ebb546569fc8db843cce58e4/users/801739c66bbd4ca4b0fac1215c736a30/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:32.742312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a08fa28d7374a029062132a73bb8ab6, user_id=b62e31d5979c426fb5b6416f087692ff, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.744571 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.744847 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.745390 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.745552 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f23bdf3-1312-4aff-a2d9-793d479dd065 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.761373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1403/2823] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:32 2022] PUT /identity/v3/projects/8a08fa28d7374a029062132a73bb8ab6/users/b62e31d5979c426fb5b6416f087692ff/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:32.767117 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.767425 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.767630 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.777159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] MFA Rules not processed for user `801739c66bbd4ca4b0fac1215c736a30`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.778516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.790127 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:32.791585 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.853653 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1421/2824] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:32.858811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.876972 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1404/2825] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:32.889896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.891896 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:32.947441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde1d65-fca7-45e3-b5d8-b55ac5e3eb8f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.952379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.952997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.953275 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.953545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.954184 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.956329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:32.971839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7fa310-8809-4dc3-9c8e-66d1176280f5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:32.976328 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:32.976937 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:32.977212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:32.977549 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:32.987819 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1422/2826] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/projects => generated 397 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:32.992900 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:32.993126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:32.995152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.007228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1405/2827] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:33.061087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:33.081701 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ecd3f-d99d-49fb-b954-13e2a38b1f05 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.089928 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.091012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.091497 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.091985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.093327 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.097743 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.136857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1423/2828] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:32 2022] POST /identity/v3/users => generated 305 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:33.141754 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.166680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d3eb83-0572-4c80-a20c-69344aa0e3f4 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.172502 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.173621 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.173853 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.174471 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.175703 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.177556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.194580 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1406/2829] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:33.199499 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.218792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dfede-5304-4de8-80e3-69fe5dcce5aa admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.227462 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.228695 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.229176 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.229731 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] PATH_INFO: `/v3/projects/a22d55e4da1340859604e4401f35d928/users/ba9db7a33cbf4099b3034cce3dcd3664/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.259309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a22d55e4da1340859604e4401f35d928, user_id=ba9db7a33cbf4099b3034cce3dcd3664, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.262089 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.282914 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1424/2830] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:33 2022] PUT /identity/v3/projects/a22d55e4da1340859604e4401f35d928/users/ba9db7a33cbf4099b3034cce3dcd3664/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:33.287477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.299408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8641dec6-edb4-4381-87b0-a9bbaba4fe0c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.341359 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:33.396689 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.397643 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.398103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.398462 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.399708 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.402835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6174c96d-c6e7-4183-95a9-bda6b233d9b0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.418739 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1407/2831] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:33.427890 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.472432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c61e2-ec9b-4df0-8be3-29becd4eef28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.478240 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.478968 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.479484 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.479867 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] PATH_INFO: `/v3/projects/a22d55e4da1340859604e4401f35d928/users/ba9db7a33cbf4099b3034cce3dcd3664/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.482173 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a22d55e4da1340859604e4401f35d928, user_id=ba9db7a33cbf4099b3034cce3dcd3664, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.485773 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.498379 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1425/2832] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:33 2022] PUT /identity/v3/projects/a22d55e4da1340859604e4401f35d928/users/ba9db7a33cbf4099b3034cce3dcd3664/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:33.510787 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.511083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.511362 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.550845 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] MFA Rules not processed for user `ba9db7a33cbf4099b3034cce3dcd3664`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:33.552888 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:33.646247 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1408/2833] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:33 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:33.654051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.655929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:33.724480 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfaff3f-8920-426a-b40f-de88abdd3679 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.729605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.730235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.730583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.730809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.743311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.744835 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8407bcf-4796-4b8c-9354-7c0b79da5187 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.757161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1426/2834] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:33 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:33.849970 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:33.860883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4810a135-5b15-4a3b-ba08-f38fe0e4fabe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:33.866106 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:33.866816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:33.867088 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:33.867542 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:33.959541 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:33.961439 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:33.976775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1409/2835] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:33 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:38.274528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.275145 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.275310 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.312136 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] MFA Rules not processed for user `b62e31d5979c426fb5b6416f087692ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:38.313386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:38.419870 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1427/2836] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:38.426194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.499330 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:38.518208 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f73559-273c-4adf-8982-00e5473c1b5d service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:38.523523 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:38.524251 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.524525 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.524791 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.525455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:38.527452 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:38.582932 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1410/2837] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:38 2022] POST /identity/v3/projects => generated 385 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:38.583986 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.597113 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd9a28f-c2bf-4c83-a516-94afece9a815 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:38.640652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:38.673474 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:38.674761 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.675209 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.675682 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.676417 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:38.678605 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:38.716456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1428/2838] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:38 2022] POST /identity/v3/users => generated 299 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:38.722677 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.740175 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a688b8-54cb-439b-8d50-39d659978934 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:38.747873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:38.748556 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.748839 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.749140 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.750023 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:38.752755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:38.769100 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1411/2839] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:38.774185 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.800690 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a58ffc9-3c53-40c7-a482-e751ebef3044 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:38.816524 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:38.819277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.820324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.820797 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] PATH_INFO: `/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb/users/1037536494e84f0b80cf05d6ba23848b/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.849746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.851926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=58217ece784949dc8ae4d6f2cb9b02fb, user_id=1037536494e84f0b80cf05d6ba23848b, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:38.857623 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:38.869320 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dc2f3-2845-4f6c-92e7-24908b5e0131 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:38.879426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:38.880599 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:38.881076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:38.881426 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] PATH_INFO: `/v3/users/801739c66bbd4ca4b0fac1215c736a30` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:38.883142 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=801739c66bbd4ca4b0fac1215c736a30)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:38.885418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:38.886593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1429/2840] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:38 2022] PUT /identity/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb/users/1037536494e84f0b80cf05d6ba23848b/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:38.893482 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:38.990667 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '801739c66bbd4ca4b0fac1215c736a30'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:39.000234 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29152482-0725-420e-b6d3-534fa508c8cc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:39.006862 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '801739c66bbd4ca4b0fac1215c736a30'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:39.010040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:39.011177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:39.011675 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:39.012158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:39.013573 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:39.016919 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:39.024774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '801739c66bbd4ca4b0fac1215c736a30'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:39.026098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '801739c66bbd4ca4b0fac1215c736a30'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:39.027064 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ad3c7a33-0a3d-4fd7-9509-6cf4c39837dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '801739c66bbd4ca4b0fac1215c736a30'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:39.032101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1430/2841] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:38 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:39.037745 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:39.040182 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:39.052076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1412/2842] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:38 2022] DELETE /identity/v3/users/801739c66bbd4ca4b0fac1215c736a30 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:39.055516 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee881f5-8200-4082-9ac1-3d79ab30d57f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:39.059554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:39.060030 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:39.060517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:39.090797 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:39.092055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:39.136319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:39.137021 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:39.137299 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:39.137574 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] PATH_INFO: `/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb/users/1037536494e84f0b80cf05d6ba23848b/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:39.139823 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] RBAC: Authorizing `identity:create_grant(project_id=58217ece784949dc8ae4d6f2cb9b02fb, user_id=1037536494e84f0b80cf05d6ba23848b, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:39.142500 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:39.166203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1431/2843] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:39 2022] PUT /identity/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb/users/1037536494e84f0b80cf05d6ba23848b/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:55:39.203104 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1413/2844] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:39.211100 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:39.213027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:39.287232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:39.338322 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c159f642-99c2-408d-8911-d28462fc382b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:39.344926 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:39.347104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:39.347104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:39.347104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:39.507170 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:39.514602 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:39.556777 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1432/2845] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:40.102724 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:40.105060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:40.124839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:40.131164 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:40.176147 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cd0a63-2e4b-4819-bec9-b9d81b690037 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:40.186982 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:40.187861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:40.191702 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:40.192179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:40.192622 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] PATH_INFO: `/v3/projects/f890a618ebb546569fc8db843cce58e4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:40.195072 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f890a618ebb546569fc8db843cce58e4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:40.198859 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:40.218571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077751aa-1ef1-4588-ad00-158de8e08b6d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:40.223650 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:40.224720 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:40.224998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:40.225275 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:40.228005 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:40.272042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f890a618ebb546569fc8db843cce58e4', 'request_id': 'req-4a270a0e-7648-45c0-9330-9952975279e5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.285759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f890a618ebb546569fc8db843cce58e4', 'request_id': 'req-4a270a0e-7648-45c0-9330-9952975279e5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.298054 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f890a618ebb546569fc8db843cce58e4', 'request_id': 'req-4a270a0e-7648-45c0-9330-9952975279e5'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.316997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] The token cache is being invalidate because project f890a618ebb546569fc8db843cce58e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:40.317651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.318537 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.323825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1433/2846] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:40 2022] DELETE /identity/v3/projects/f890a618ebb546569fc8db843cce58e4 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:40.325238 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:40.327062 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:40.360795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:40.363922 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:40.382637 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1414/2847] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:40.433395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a270a0e-7648-45c0-9330-9952975279e5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:40.443985 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:40.445567 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:40.446251 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:40.447111 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] PATH_INFO: `/v3/users/ba9db7a33cbf4099b3034cce3dcd3664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:40.449407 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba9db7a33cbf4099b3034cce3dcd3664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:40.453678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:40.467188 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:40.570436 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba9db7a33cbf4099b3034cce3dcd3664'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.581881 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba9db7a33cbf4099b3034cce3dcd3664'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.607277 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba9db7a33cbf4099b3034cce3dcd3664'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.608651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba9db7a33cbf4099b3034cce3dcd3664'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.625487 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-63c52168-ef2d-4e65-83f2-c3e0de96d580 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba9db7a33cbf4099b3034cce3dcd3664'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:40.642078 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1434/2848] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:55:40 2022] DELETE /identity/v3/users/ba9db7a33cbf4099b3034cce3dcd3664 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:41.073311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.089658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb940b-57a2-47bf-a1c9-9142f934f8c0 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.171494 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.173228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.174835 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.175336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] PATH_INFO: `/v3/projects/a22d55e4da1340859604e4401f35d928` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.177781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] RBAC: Authorizing `identity:delete_project(project_id=a22d55e4da1340859604e4401f35d928)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.180074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.296051 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a22d55e4da1340859604e4401f35d928', 'request_id': 'req-1360c33b-b0a1-4ed9-b036-debee84a49da'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:41.304824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a22d55e4da1340859604e4401f35d928', 'request_id': 'req-1360c33b-b0a1-4ed9-b036-debee84a49da'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:41.316958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a22d55e4da1340859604e4401f35d928', 'request_id': 'req-1360c33b-b0a1-4ed9-b036-debee84a49da'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:41.337919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] The token cache is being invalidate because project a22d55e4da1340859604e4401f35d928 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:55:41.338152 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:41.339164 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1360c33b-b0a1-4ed9-b036-debee84a49da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:55:41.340815 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1415/2849] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:55:41 2022] DELETE /identity/v3/projects/a22d55e4da1340859604e4401f35d928 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:41.347233 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.348295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.348759 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.378093 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:41.379799 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:41.474903 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1435/2850] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:55:41.483450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.483731 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.484071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.503183 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:41.510645 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:41.513228 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:41.525548 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1416/2851] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:41.530177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.542141 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9813b5c4-fc2a-4b4f-af7d-8f9f262a53c4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.546950 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.547749 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.547973 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.548244 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.549660 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:55:41.549993 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.552041 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.566659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1436/2852] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:55:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:55:41.571828 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.582453 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90350b78-8ce0-459e-b030-c2d3c9c6571b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.587684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.588373 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.588589 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.588861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.589510 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.592011 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.648959 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.649178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1417/2853] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:41 2022] POST /identity/v3/projects => generated 373 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:41.663805 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7351966-3356-42ae-acbd-0bf14175d2d4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.775848 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.776884 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.777360 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.777849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.779035 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.782893 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.822583 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1437/2854] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:41 2022] POST /identity/v3/users => generated 287 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:41.826971 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.841464 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1008ac35-945f-474a-9a4a-7f252fc812ff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.850349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.852864 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.853371 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.854836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.856068 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.858786 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.881000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1418/2855] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:55:41.883333 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.894282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:41.905219 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20fa955-203b-4b7c-aff9-1aae8520595a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:41.926934 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:41.928196 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:41.928712 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:41.929205 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:41.930336 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:41.933861 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3c6d53-b9d4-4048-835e-93688e515a9c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:41.964104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1438/2856] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:41.970381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:41.998665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4ebf7-cd94-4ef7-8b38-46bc04f9af12 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:42.008758 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:42.009973 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:42.010467 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:42.010988 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] PATH_INFO: `/v3/projects/e4453eddd61d45c9a8007ba49656cff7/users/360dfb2e78b342eb8426c887b2caa724/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:42.031243 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4453eddd61d45c9a8007ba49656cff7, user_id=360dfb2e78b342eb8426c887b2caa724, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:42.035388 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:42.054994 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1419/2857] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:41 2022] PUT /identity/v3/projects/e4453eddd61d45c9a8007ba49656cff7/users/360dfb2e78b342eb8426c887b2caa724/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:42.059001 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3260ac0c-4eef-4d99-82b5-d85f73f4e9fa None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:42.059703 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3260ac0c-4eef-4d99-82b5-d85f73f4e9fa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:42.060101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3260ac0c-4eef-4d99-82b5-d85f73f4e9fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:42.084748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3260ac0c-4eef-4d99-82b5-d85f73f4e9fa None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:42.086346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3260ac0c-4eef-4d99-82b5-d85f73f4e9fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:42.219115 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1439/2858] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:42.237481 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:42.239685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:42.350861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74928947-8094-45eb-b0ce-3cb29b95c8ee admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:42.355891 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:42.356680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:42.356965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:42.357236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:42.369433 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:42.371010 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:42.385412 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1420/2859] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:44.281450 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:44.281935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:44.282065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:44.339490 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] MFA Rules not processed for user `360dfb2e78b342eb8426c887b2caa724`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:44.341049 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:44.514091 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1440/2860] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:44.519513 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:44.539387 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d5136-2e0f-494f-8892-0f2014d16d0c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:44.546194 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:44.547167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:44.547717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:44.548170 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:44.549312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:44.551744 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:44.617313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1421/2861] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:55:44 2022] POST /identity/v3/projects => generated 371 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:55:44.621758 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:44.647270 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c423376-87cb-4307-bf23-ce8b8527039b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:44.750931 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:44.753162 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:44.754730 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:44.755019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:44.755781 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:44.757811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:44.793518 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1441/2862] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:55:44 2022] POST /identity/v3/users => generated 292 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:55:44.797296 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:44.813459 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a002e-1b56-4564-8f63-3e280cf413ab admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:44.824031 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:44.824554 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:44.824837 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:44.825114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:44.825795 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:44.827544 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:44.847056 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1422/2863] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:44.851617 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:44.882769 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be40fa24-b926-4fb3-8dbf-2e19971b1d28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:44.916388 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:44.917689 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:44.918885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:44.919195 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] PATH_INFO: `/v3/projects/29ac56af9fae47e19e8b931c93a6e145/users/5b86413843954fe5be1e90022a80c97f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:44.939708 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:44.952189 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29ac56af9fae47e19e8b931c93a6e145, user_id=5b86413843954fe5be1e90022a80c97f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:44.961818 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:44.997822 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1442/2864] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:44 2022] PUT /identity/v3/projects/29ac56af9fae47e19e8b931c93a6e145/users/5b86413843954fe5be1e90022a80c97f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:45.001776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:45.014321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dd193-527f-43bf-bf8f-305f2aaaff8a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:45.137238 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:45.138669 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:45.139155 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:45.139721 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:45.146583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:45.146583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db05b0d-76f2-4368-afa1-37126a1df22c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:45.167159 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1423/2865] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:55:45.171150 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:45.185963 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a16df0-e9a1-45f7-83dd-ed6e681d3772 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:45.194849 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:45.195929 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:45.197910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:45.203213 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] PATH_INFO: `/v3/projects/29ac56af9fae47e19e8b931c93a6e145/users/5b86413843954fe5be1e90022a80c97f/roles/d7f1381a769e433aa1b033022931be96` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:45.207203 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29ac56af9fae47e19e8b931c93a6e145, user_id=5b86413843954fe5be1e90022a80c97f, role_id=d7f1381a769e433aa1b033022931be96)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:45.212402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c949383e-3de6-4b9f-bb86-050b0b531070 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:45.230492 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1443/2866] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:55:45 2022] PUT /identity/v3/projects/29ac56af9fae47e19e8b931c93a6e145/users/5b86413843954fe5be1e90022a80c97f/roles/d7f1381a769e433aa1b033022931be96 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:55:46.085206 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:46.085841 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:46.086216 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:46.145873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:46.176803 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] MFA Rules not processed for user `1037536494e84f0b80cf05d6ba23848b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:46.178443 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.248194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:46.248981 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:46.249314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:46.297264 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] MFA Rules not processed for user `5b86413843954fe5be1e90022a80c97f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:46.300155 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.313354 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1424/2867] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:46 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:46.327883 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:46.329283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.396737 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ecca7f-0ab6-4ff7-a31e-2137355df1f3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:46.401055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:46.401826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:46.402177 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1444/2868] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:46 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:46.402575 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:46.402909 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:46.405072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.414456 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:46.416325 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.474308 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:46.475816 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211847d-0562-4ffa-a252-064ece3990c7 service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:46.483742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef27190-2cf2-441a-bb67-cfaf55521fc4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:46.486343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1425/2869] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:46.488523 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:46.489094 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:46.489725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:46.489725 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:46.556936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:46.558776 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:46.572821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1445/2870] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:55:46.672625 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:46.673317 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:46.673492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:46.741239 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:46.742478 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.774511 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:46.854259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1426/2871] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:55:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:46.923403 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:46.926229 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:46.947587 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:55:47.064714 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fda1c-aefb-4c2d-86d8-b0d04c015092 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:47.070783 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:47.071509 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:47.071742 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:47.072016 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:47.083099 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:47.084718 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172a26d3-bdda-4ae8-b7ff-37d3c9548ec9 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:47.094190 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1446/2872] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:55:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:55:48.085348 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:48.087007 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:48.100356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869df172-9734-4ae7-89a1-0022b00dc9f6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:48.105734 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:48.106353 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:48.106658 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:48.106974 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:48.124072 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:48.127111 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:48.139336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1427/2873] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:48 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:56.678419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e8548-0b46-4f99-87dd-0e4c1e8243aa None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:56.679019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e8548-0b46-4f99-87dd-0e4c1e8243aa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:56.679019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092e8548-0b46-4f99-87dd-0e4c1e8243aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:56.699739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-092e8548-0b46-4f99-87dd-0e4c1e8243aa None None] MFA Rules not processed for user `b62e31d5979c426fb5b6416f087692ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:56.700998 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-092e8548-0b46-4f99-87dd-0e4c1e8243aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:56.708200 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1447/2874] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:55:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:57.042200 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:57.043775 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:57.103501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b62b2f-d59a-4d61-8866-3fbb4d58c59b service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:57.107765 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:57.108370 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:57.108631 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:57.108923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:57.111094 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:57.123067 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:57.124680 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:57.134865 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1428/2875] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:55:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:55:58.830713 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:58.831309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:58.831659 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:58.852779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:55:58.853994 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:55:58.861862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1448/2876] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:55:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:55:58.874156 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:55:58.887040 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b59bda5-0a07-457f-b537-fd69424954bc service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:55:58.892311 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:55:58.892939 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:55:58.893217 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:55:58.893487 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:55:58.902263 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:55:58.903846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:55:58.913017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1429/2877] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:55:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:56:02.730921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:02.826481 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cf9241-ae1b-4dcd-a58c-8af223e984bb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:02.832752 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:02.833402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:02.833692 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:02.833935 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] PATH_INFO: `/v3/users/360dfb2e78b342eb8426c887b2caa724` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:02.836331 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=360dfb2e78b342eb8426c887b2caa724)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:02.837358 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:02.908549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '360dfb2e78b342eb8426c887b2caa724'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:02.917610 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '360dfb2e78b342eb8426c887b2caa724'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:02.934593 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:02.937406 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '360dfb2e78b342eb8426c887b2caa724'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:02.937824 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '360dfb2e78b342eb8426c887b2caa724'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:02.948739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '360dfb2e78b342eb8426c887b2caa724'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:02.956127 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1449/2878] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:56:02 2022] DELETE /identity/v3/users/360dfb2e78b342eb8426c887b2caa724 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:03.235062 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:03.272000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fef793d-1361-4274-8e1c-6e7c237dd432 service swift] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:03.367886 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:03.368480 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:03.368755 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:03.369026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] PATH_INFO: `/v3/projects/e4453eddd61d45c9a8007ba49656cff7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:03.370283 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4453eddd61d45c9a8007ba49656cff7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:03.372450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:03.451830 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:03.462579 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4453eddd61d45c9a8007ba49656cff7', 'request_id': 'req-d01986a1-197d-4738-a620-c387e8909d5a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.471484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4453eddd61d45c9a8007ba49656cff7', 'request_id': 'req-d01986a1-197d-4738-a620-c387e8909d5a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.490259 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4453eddd61d45c9a8007ba49656cff7', 'request_id': 'req-d01986a1-197d-4738-a620-c387e8909d5a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.510092 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] The token cache is being invalidate because project e4453eddd61d45c9a8007ba49656cff7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:56:03.510568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.511466 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.514047 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1430/2879] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:56:03 2022] DELETE /identity/v3/projects/e4453eddd61d45c9a8007ba49656cff7 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:03.524455 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:03.526202 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:03.551599 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ac593a-be15-4b3b-85bb-2ce2e2478d34 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:03.646320 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:03.647011 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:03.647282 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:03.647549 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] PATH_INFO: `/v3/users/5b86413843954fe5be1e90022a80c97f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:03.648661 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b86413843954fe5be1e90022a80c97f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:03.650748 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:03.712607 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b86413843954fe5be1e90022a80c97f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.720536 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b86413843954fe5be1e90022a80c97f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.740910 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b86413843954fe5be1e90022a80c97f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.741954 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b86413843954fe5be1e90022a80c97f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.752303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-26276a3c-94ab-42de-aa69-04fd640236f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b86413843954fe5be1e90022a80c97f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:03.759329 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1450/2880] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:56:03 2022] DELETE /identity/v3/users/5b86413843954fe5be1e90022a80c97f => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:03.953855 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:03.955991 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:03.972634 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01986a1-197d-4738-a620-c387e8909d5a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.061114 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.061873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.062189 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.062491 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] PATH_INFO: `/v3/projects/29ac56af9fae47e19e8b931c93a6e145` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.063804 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29ac56af9fae47e19e8b931c93a6e145)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.065933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.133959 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29ac56af9fae47e19e8b931c93a6e145', 'request_id': 'req-2bf7e970-d22a-4a6b-a146-1f738f46f106'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:04.143301 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29ac56af9fae47e19e8b931c93a6e145', 'request_id': 'req-2bf7e970-d22a-4a6b-a146-1f738f46f106'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:04.165619 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29ac56af9fae47e19e8b931c93a6e145', 'request_id': 'req-2bf7e970-d22a-4a6b-a146-1f738f46f106'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:04.186910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] The token cache is being invalidate because project 29ac56af9fae47e19e8b931c93a6e145 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:56:04.186910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:04.194921 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2bf7e970-d22a-4a6b-a146-1f738f46f106 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:04.198408 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1431/2881] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:56:03 2022] DELETE /identity/v3/projects/29ac56af9fae47e19e8b931c93a6e145 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:04.208006 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.211560 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.215873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.256311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:04.257875 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:04.355999 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1451/2882] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:56:04.364492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.365071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.365071 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.388181 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:04.389418 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:04.401953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1432/2883] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:04.406441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.417936 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17537d92-ad3d-43a5-ae95-e01d5fd221eb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.432977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.432977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.432977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.432977 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.433608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:56:04.433608 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.435458 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.439274 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:04.451346 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1452/2884] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:56:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:56:04.457958 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.470236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beacfb93-b5c2-480f-918e-7cd7df74436d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.475209 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.476538 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.476849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.477096 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.477746 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.479912 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.518666 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1433/2885] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:56:04 2022] POST /identity/v3/projects => generated 369 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:56:04.520909 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.543569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a876e2a-13b4-49db-a580-f70a72dcbddb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.623224 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.624104 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.624382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.624665 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.625297 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.628003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.662309 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1453/2886] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:56:04 2022] POST /identity/v3/users => generated 285 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 04 08:56:04.667509 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.684054 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24fef00-2a8b-4235-b373-7570f9226877 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.698517 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.700404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.700925 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.701385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.702717 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.708811 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.743824 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1434/2887] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:56:04.745271 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.777693 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff79416b-001c-486e-91f1-db9e440f7e5c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.785694 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.786159 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.786442 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.786739 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.787402 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.789051 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f706cac4-729b-4b6a-a93e-567335659529 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.817119 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1454/2888] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:56:04.825520 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:04.837174 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e10cd61-d61e-4f48-abd4-bafc89793b06 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:04.842903 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:04.843659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.843998 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.844332 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] PATH_INFO: `/v3/projects/a6041b2caedc4b28a67d1ed12e4b9787/users/b31b32330ccb413daa184ea5d5f6379c/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:04.875012 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6041b2caedc4b28a67d1ed12e4b9787, user_id=b31b32330ccb413daa184ea5d5f6379c, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:04.879167 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:04.881654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:04.928684 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1435/2889] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:56:04 2022] PUT /identity/v3/projects/a6041b2caedc4b28a67d1ed12e4b9787/users/b31b32330ccb413daa184ea5d5f6379c/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:04.985866 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:04.986126 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:04.986395 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:05.017419 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] MFA Rules not processed for user `b31b32330ccb413daa184ea5d5f6379c`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:05.018678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:05.092266 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1455/2890] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:04 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:05.105102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:05.106919 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:05.176445 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79946699-7e95-4ba1-81b1-888a1fafac95 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:05.180325 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbc7631562d645dd9b7876a59de53a31', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:05.180950 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:05.181282 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:05.181492 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:05.191753 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:05.198497 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:05.207098 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1436/2891] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:05 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:56:15.088857 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=68,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:15.200118 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=1,keystone:SELECT=28 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:25.744812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:25.746121 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:25.868440 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4991173-4610-4600-8d5c-68a9be78dc3d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:25.872858 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:25.873577 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:25.874027 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:25.874324 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:25.876386 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:25.889156 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:25.890871 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:25.900469 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1456/2892] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 04 08:56:32.047761 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:32.049975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:32.072725 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd259b0d-e425-44b2-ba86-1fef1a92eedd service glance] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:32.076455 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:32.077055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:32.077336 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:32.077639 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:32.079399 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:32.155077 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:32.156594 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb22ab93-93a6-4ee3-8b2d-bbc5960c930f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:32.168776 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1437/2893] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:32 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:56:35.896306 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=35 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:36.094153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:36.172918 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5281def8-40d5-4b33-8c9d-dd1df9628a9e service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:36.180087 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:36.181012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:36.181300 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:36.181569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] PATH_INFO: `/v3/users/b31b32330ccb413daa184ea5d5f6379c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:36.182816 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b31b32330ccb413daa184ea5d5f6379c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:36.185697 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:36.247812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b31b32330ccb413daa184ea5d5f6379c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:36.257809 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b31b32330ccb413daa184ea5d5f6379c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:36.274013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b31b32330ccb413daa184ea5d5f6379c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:36.275020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b31b32330ccb413daa184ea5d5f6379c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:36.292132 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b31b32330ccb413daa184ea5d5f6379c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:36.301003 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1457/2894] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:56:36 2022] DELETE /identity/v3/users/b31b32330ccb413daa184ea5d5f6379c => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:36.305570 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:36.306102 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:36.306289 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:36.330415 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:36.332312 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:36.402212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1438/2895] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 04 08:56:36.412019 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:36.413539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:36.478673 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b307833-9988-47da-8ea1-8becbe74b8c3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:36.482737 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:36.483350 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:36.483625 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:36.483895 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:36.494997 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:36.496494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8c8cf6-36ec-4c7b-be45-3ff13f6955e3 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:36.508160 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1458/2896] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:56:36.983975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:36.985427 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:37.000132 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d20d69-e17b-4fce-857a-736739802cfe None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.006450 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.007109 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.007380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.008019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] PATH_INFO: `/v3/projects/a6041b2caedc4b28a67d1ed12e4b9787` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.010158 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6041b2caedc4b28a67d1ed12e4b9787)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.012444 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.062029 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6041b2caedc4b28a67d1ed12e4b9787', 'request_id': 'req-63400e57-f119-4241-a3a9-a95de634befc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:37.068772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6041b2caedc4b28a67d1ed12e4b9787', 'request_id': 'req-63400e57-f119-4241-a3a9-a95de634befc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:37.076757 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6041b2caedc4b28a67d1ed12e4b9787', 'request_id': 'req-63400e57-f119-4241-a3a9-a95de634befc'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:37.087930 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] The token cache is being invalidate because project a6041b2caedc4b28a67d1ed12e4b9787 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:56:37.088212 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:37.088836 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-63400e57-f119-4241-a3a9-a95de634befc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:56:37.090314 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1439/2897] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:56:36 2022] DELETE /identity/v3/projects/a6041b2caedc4b28a67d1ed12e4b9787 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:56:37.095172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.095691 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.095812 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.116018 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:37.117197 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:37.135046 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:37.181528 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1459/2898] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:37.189331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.189609 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.190409 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.204606 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:37.211953 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:37.214055 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:37.222849 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1440/2899] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:37.227393 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.237004 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259553ae-89b5-4a64-a49d-1115efcebb20 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.242101 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.242740 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.243024 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.243295 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.243938 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 04 08:56:37.244236 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.245825 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.258432 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1460/2900] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Oct 4 08:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 04 08:56:37.263017 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.273512 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf873b95-7352-40e1-a508-9e4cb697b442 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.278236 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.278863 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.279144 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.279381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.280006 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.281870 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.308975 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1441/2901] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Oct 4 08:56:37 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:56:37.313149 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.323882 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea3280-8d69-4a6f-a175-8af0d4c00a27 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.380438 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.381040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.381314 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.381584 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.382212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.384153 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.405975 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1461/2902] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Oct 4 08:56:37 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 04 08:56:37.410664 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.423161 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d114d0-685d-46f5-a325-d64e15b322f6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.428723 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.429321 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.429592 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.429857 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.430501 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.432349 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.443044 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1442/2903] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 04 08:56:37.447305 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.457947 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2eb809-dac3-42f1-88b0-46ed50d08752 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.462678 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.463272 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.463545 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.463811 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.464446 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.466020 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b90f12-b2ad-4e3e-afb7-87aee1ecc47d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.478507 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1462/2904] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Oct 4 08:56:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 04 08:56:37.482638 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.494022 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe7e68d-80ed-4728-a259-2781a8936ee8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.498527 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.499178 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.499473 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.499718 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] PATH_INFO: `/v3/projects/756737695aa644ff8a9ec362b654cf3f/users/7ef96a2e81f2404f960b8882659da9c4/roles/a0ac2c0678b9462799a770a2b667dd0e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.515904 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] RBAC: Authorizing `identity:create_grant(project_id=756737695aa644ff8a9ec362b654cf3f, user_id=7ef96a2e81f2404f960b8882659da9c4, role_id=a0ac2c0678b9462799a770a2b667dd0e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.518905 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.532270 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1443/2905] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Oct 4 08:56:37 2022] PUT /identity/v3/projects/756737695aa644ff8a9ec362b654cf3f/users/7ef96a2e81f2404f960b8882659da9c4/roles/a0ac2c0678b9462799a770a2b667dd0e => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 04 08:56:37.543873 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf6a74-6d1c-4711-b489-4f22c7193d0f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.544161 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf6a74-6d1c-4711-b489-4f22c7193d0f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.544435 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf6a74-6d1c-4711-b489-4f22c7193d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.573140 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c0bf6a74-6d1c-4711-b489-4f22c7193d0f None None] MFA Rules not processed for user `7ef96a2e81f2404f960b8882659da9c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:37.574353 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c0bf6a74-6d1c-4711-b489-4f22c7193d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:37.641808 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1463/2906] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:37.652874 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:37.654654 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:37.714697 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793c71c-7811-4a99-8a18-b9c019df2240 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:37.718313 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '546c67695c1e447da68b91637aefe135', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:37.718938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:37.719210 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:37.719475 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:37.727861 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:37.729319 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c7638-c637-411c-8eb4-b34534399723 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:37.739149 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1444/2907] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:56:37 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:56:39.107789 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:39.108326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:39.108326 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:39.127885 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] MFA Rules not processed for user `7ef96a2e81f2404f960b8882659da9c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:39.129084 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:39.136631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1464/2908] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:56:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:47.734000 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=50,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:49.124279 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:56:57.556595 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd943055-c7bd-468b-b6c7-863df8d5c839 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:57.557110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd943055-c7bd-468b-b6c7-863df8d5c839 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:57.557110 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd943055-c7bd-468b-b6c7-863df8d5c839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:57.576543 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fd943055-c7bd-468b-b6c7-863df8d5c839 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:57.577729 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd943055-c7bd-468b-b6c7-863df8d5c839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:57.632923 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1445/2909] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:56:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:57.673643 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:57.675081 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:57.745158 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c244a2cb-784d-4b8e-a8a2-a966e6b7ac90 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:57.750013 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:57.750642 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:57.750915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:57.751246 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:57.759836 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:57.761404 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:57.769381 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1465/2910] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:56:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:56:58.288080 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ea9f4-a5b6-407e-be7d-97bb46f1e5f4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:58.288356 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ea9f4-a5b6-407e-be7d-97bb46f1e5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:58.288622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ea9f4-a5b6-407e-be7d-97bb46f1e5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:58.308751 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bf9ea9f4-a5b6-407e-be7d-97bb46f1e5f4 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:58.309969 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bf9ea9f4-a5b6-407e-be7d-97bb46f1e5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:58.317762 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1446/2911] 10.222.0.16 () {58 vars in 967 bytes} [Tue Oct 4 08:56:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:58.330139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:58.342065 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8bea69-01db-48e4-96d7-36aa2ddee44f service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:58.347079 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:58.347669 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:58.347940 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:58.348255 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:58.357042 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:58.358631 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:58.367525 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1466/2912] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:56:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:56:58.603535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a7abe5-b8b2-471a-8979-81064f899acc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:58.604019 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a7abe5-b8b2-471a-8979-81064f899acc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:58.604128 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a7abe5-b8b2-471a-8979-81064f899acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:58.624381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f6a7abe5-b8b2-471a-8979-81064f899acc None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:58.625568 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f6a7abe5-b8b2-471a-8979-81064f899acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:58.633774 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1447/2913] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Oct 4 08:56:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:58.645790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:58.656235 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f4efc-02fa-4e55-ad17-d197d4d5d7e1 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:58.661344 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:58.661944 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:58.662223 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:58.662494 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:58.670433 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:58.672026 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:58.681012 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1467/2914] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:56:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:56:59.009979 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78f491-da76-4884-8611-163e1603892c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:59.010261 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78f491-da76-4884-8611-163e1603892c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:59.010534 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78f491-da76-4884-8611-163e1603892c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:59.030873 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8a78f491-da76-4884-8611-163e1603892c None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:56:59.032083 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8a78f491-da76-4884-8611-163e1603892c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:56:59.039449 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1448/2915] 10.222.0.16 () {58 vars in 967 bytes} [Tue Oct 4 08:56:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:56:59.051792 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:56:59.062995 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbbd860-4307-4b91-adf0-7e1799c0a296 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:56:59.069218 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:56:59.069927 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:56:59.070125 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:56:59.070533 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:56:59.079179 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:56:59.080750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db302ac-2b1d-4f6f-90ae-bed43b12484f service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:56:59.089312 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1468/2916] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:56:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:57:01.206622 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78f491-da76-4884-8611-163e1603892c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:01.272026 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78f491-da76-4884-8611-163e1603892c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:01.276693 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:01.277340 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:01.277600 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:01.277878 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] PATH_INFO: `/v3/users/7ef96a2e81f2404f960b8882659da9c4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:01.279074 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ef96a2e81f2404f960b8882659da9c4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:01.280987 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:01.327846 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ef96a2e81f2404f960b8882659da9c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.335305 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ef96a2e81f2404f960b8882659da9c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.346954 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ef96a2e81f2404f960b8882659da9c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.347646 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ef96a2e81f2404f960b8882659da9c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.348805 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ef96a2e81f2404f960b8882659da9c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.361477 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1449/2917] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:57:01 2022] DELETE /identity/v3/users/7ef96a2e81f2404f960b8882659da9c4 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:01.367073 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:01.367539 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:01.367663 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:01.386340 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] MFA Rules not processed for user `a17a7c6f52b74569b1779d66c4e1a9d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:01.387600 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:01.445821 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1469/2918] 10.222.0.16 () {58 vars in 958 bytes} [Tue Oct 4 08:57:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:01.455358 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:01.456781 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:01.464060 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:01.515675 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37605566-4047-49aa-ab70-d282ea0c5da8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:01.519295 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3e66140a80f641f4afe94ed0452000f8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:01.519887 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:01.520690 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:01.520952 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:01.530381 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:01.531826 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effbde16-3027-4187-8651-0f2375422b79 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:01.541685 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1450/2919] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:57:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:57:01.714370 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:01.716139 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:01.730045 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee167f1d-2e5d-4e0e-a5e5-feb3c854755c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:01.734741 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:01.735382 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:01.735645 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:01.735911 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] PATH_INFO: `/v3/projects/756737695aa644ff8a9ec362b654cf3f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:01.737040 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=756737695aa644ff8a9ec362b654cf3f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:01.739565 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:01.787422 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:01.791839 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '756737695aa644ff8a9ec362b654cf3f', 'request_id': 'req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.798734 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '756737695aa644ff8a9ec362b654cf3f', 'request_id': 'req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.806212 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '756737695aa644ff8a9ec362b654cf3f', 'request_id': 'req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.817451 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] The token cache is being invalidate because project 756737695aa644ff8a9ec362b654cf3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:57:01.817790 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.818820 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:01.820564 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1470/2920] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:57:01 2022] DELETE /identity/v3/projects/756737695aa644ff8a9ec362b654cf3f => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:11.537583 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=24 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:11.821241 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:44.934682 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:44.935235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:44.935235 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:44.956726 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] MFA Rules not processed for user `908ed003368445ae8db242549c212981`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 04 08:57:44.957938 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:45.015018 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1451/2921] 10.222.0.16 () {58 vars in 967 bytes} [Tue Oct 4 08:57:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 04 08:57:45.028441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:45.029876 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:45.098311 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5063bf9a-3ad4-4e57-814f-a9ca975f0b5c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:45.103779 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cc208be4d444563929a03634cfddee7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:45.104378 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:45.104652 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:45.104915 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:45.113214 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:45.115426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:45.123252 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1471/2922] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Oct 4 08:57:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 04 08:57:51.662852 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:51.666884 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:51.752331 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446e56b3-0a14-4a57-b4ce-31601d0eb605 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:51.758221 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:51.758910 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:51.759187 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:51.759457 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] PATH_INFO: `/v3/users/b62e31d5979c426fb5b6416f087692ff` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:51.760659 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=b62e31d5979c426fb5b6416f087692ff)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:51.762642 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:51.822660 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b62e31d5979c426fb5b6416f087692ff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:51.830343 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b62e31d5979c426fb5b6416f087692ff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:51.845337 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b62e31d5979c426fb5b6416f087692ff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:51.846322 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b62e31d5979c426fb5b6416f087692ff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:51.846965 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b62e31d5979c426fb5b6416f087692ff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:51.860785 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1452/2923] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:57:51 2022] DELETE /identity/v3/users/b62e31d5979c426fb5b6416f087692ff => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:52.043688 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:52.055862 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a9a45-74a2-4354-8d85-7bfaffa0f5df service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:52.118569 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:52.119302 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:52.119580 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:52.119850 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] PATH_INFO: `/v3/projects/8a08fa28d7374a029062132a73bb8ab6` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:52.120972 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a08fa28d7374a029062132a73bb8ab6)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:52.122921 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:52.178031 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a08fa28d7374a029062132a73bb8ab6', 'request_id': 'req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.184232 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a08fa28d7374a029062132a73bb8ab6', 'request_id': 'req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.191733 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a08fa28d7374a029062132a73bb8ab6', 'request_id': 'req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.202319 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] The token cache is being invalidate because project 8a08fa28d7374a029062132a73bb8ab6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:57:52.202655 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.203285 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.204786 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1472/2924] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:57:52 2022] DELETE /identity/v3/projects/8a08fa28d7374a029062132a73bb8ab6 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:52.209665 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:52.211143 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:52.224709 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e248657-2c14-4cae-bf87-52934704f2cf admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:52.244535 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:52.283165 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:52.283796 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:52.284076 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:52.284344 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] PATH_INFO: `/v3/users/1037536494e84f0b80cf05d6ba23848b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:52.285484 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1037536494e84f0b80cf05d6ba23848b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:52.288138 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:52.333037 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1037536494e84f0b80cf05d6ba23848b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.340458 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1037536494e84f0b80cf05d6ba23848b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.353099 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1037536494e84f0b80cf05d6ba23848b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.354103 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1037536494e84f0b80cf05d6ba23848b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.354772 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1037536494e84f0b80cf05d6ba23848b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.367571 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1453/2925] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Oct 4 08:57:52 2022] DELETE /identity/v3/users/1037536494e84f0b80cf05d6ba23848b => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:57:52.544831 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:57:52.546281 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:57:52.561620 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b5ac48-b648-4f31-b588-4a7afb0f0e55 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:57:52.579750 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:57:52.622071 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a17a7c6f52b74569b1779d66c4e1a9d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc7bdc31e1404eecb2eaeb287aaeab06', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:57:52.622651 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:57:52.622930 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:57:52.623199 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] PATH_INFO: `/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:57:52.624303 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58217ece784949dc8ae4d6f2cb9b02fb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:57:52.626194 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:57:52.674172 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58217ece784949dc8ae4d6f2cb9b02fb', 'request_id': 'req-87fc57d3-d580-4239-97cb-1f47df84d6d6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.680426 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58217ece784949dc8ae4d6f2cb9b02fb', 'request_id': 'req-87fc57d3-d580-4239-97cb-1f47df84d6d6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.687719 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58217ece784949dc8ae4d6f2cb9b02fb', 'request_id': 'req-87fc57d3-d580-4239-97cb-1f47df84d6d6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.698619 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] The token cache is being invalidate because project 58217ece784949dc8ae4d6f2cb9b02fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 04 08:57:52.698905 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.699521 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-87fc57d3-d580-4239-97cb-1f47df84d6d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 04 08:57:52.700974 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1473/2926] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Oct 4 08:57:52 2022] DELETE /identity/v3/projects/58217ece784949dc8ae4d6f2cb9b02fb => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 04 08:58:02.364176 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=42,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:02.695441 dv-cinder-860209-1-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 04 08:58:33.955933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 04 08:58:33.957378 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:34.029329 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11748ec7-276a-4c71-874a-87afe9a097e2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 04 08:58:34.089506 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '18e6e8451d334251a87f84b03e57a4c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '427c33aa73c94f0e9e383089fb5f6fb1', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 04 08:58:34.090108 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 04 08:58:34.090380 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 04 08:58:34.090681 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 04 08:58:34.092404 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 04 08:58:34.103385 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 04 08:58:34.104933 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44685fa6-a884-4476-84df-6fd07d20b69c service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 04 08:58:34.114928 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1454/2927] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Oct 4 08:58:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 04 08:58:44.110383 dv-cinder-860209-1-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=58 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}